Big Data technologies are trendy today, as evidenced by at least the fact that, at the moment, it is one of the most frequently used terms in IT publications. It is enough to look at the statistics of such well-known search engines as Google or Yandex on the phrase “Big Data”. It becomes clear that the expression “Big Data” can now be called one of the most popular areas of information technology development.

So what is the secret of these technologies’ popularity, and what does the term “Big Data” mean? First of all, the name “Big Data” means a massive set of information. Moreover, its volume is so large that it becomes extremely tough to process such amounts of data with conventional software and hardware. …

In discussions on work, I heard many stories about how bad it is to work in a particular project/company/industry, etc. Some of them are indeed valid. However, having gained quite a lot of experience working in companies and changing many projects and technology stacks, I have understood the problem and solution in different ways. We will talk about this.

When I hear complaints from employees about their dissatisfaction with their project (company/office/programming stack / etc.), I want to share my conclusions. Here they are:

We make choices in conditions of insufficient information.

Often in life, there might be situations when we need to make choices in conditions of insufficient information about options. For example, when students plan to go to college, they often have a rather vague idea of ​​available options. Personally, It was a problem to understand what educational opportunities are in general, what universities are in my city, and whether it is worth applying to them. Moreover, I did not understand which university would provide me with the most high-quality and relevant knowledge. I figured it out only after ten years. I doubt that many school graduates have such an understanding. …

It is not a secret that in many companies, expenses on salaries of IT departments’ employees make up a significant part of the budget. And the high competition in the labor market further pushes the level of wages for IT specialists. High expenses lead to the fact that the employer is always searching for options to reduce their costs. It turned out, the companies themselves, trying to reduce costs as much as possible, turn to various tricks to reduce the cost of the work done. Similar mechanisms are used not only in IT but also in different other areas of the employer. …

by Mkrtich Pudeian

Data warehouse projects have long been part of the IT infrastructure of most large companies. ETL processes are part of these projects, but developers sometimes make the same mistakes when designing and maintaining these processes. Some of these mistakes are described in this post.

I would like to immediately narrow down the scope of the discussion and agree on terminology:

• Data warehouse (Datawarehouse, DWH) means traditional SQL DWH (Oracle Database, MS SQL Server, etc.);
• When modeling DWH, the concepts of a single version of the truth and historical truth are usually implied;
• ETL-process (Extraction-Transformation-Loading) refers to the process of loading data from one or more source systems into DWH. …

Здравствуйте, дорогие друзья.

Сегодня хочу поделиться историей из жизни, как было устроено хранилище DWH в Tele2 до внедрения КХД (EDW).

Поступил я в ИТ подразделение Tele2 в 2012 в отдел по системам отчетности. На тот момент в компании уже было создано хранилище DWH, на котором уже крутилось много процессов по предоставлению отчетности и не только.

Немного по поводу технического стека, который там использовался на тот момент. Для хранилища использовалась Оракловая база объемом 60–100 Тб сервер T4–4 c оперативой под 1 Тб. Туда загружались данные из различных источников. Но основными из них были 4 оракловые биллинговые базы, которые были по сути платформой тарификации. И был отдел, который занимался поддержкой этих баз и предоставлением сервисов. Разделение этих баз было по макрорегионам. Причина: слишком большие объемы. Т.е …