It’s not nearly as boring as you may think
“Oh my!” I hear you cry… that is a very boring topic. But let me make a radical argument: a journey through monitoring is a journey through the entire evolution of IT.
For decades—from the mainframe era to now—network monitoring has played a critical role in ensuring the smooth functioning and security of computer networks, adapting along the way to ever-increasing complexity and scale. And my own career matches this history.
Mainframes
I started work in 1988 as a trainee mainframe operator, the last of the breed. The traditional mainframe was near the end of its over 30-year run. Computer networks were centralized hub-and-spoke affairs, with a large mainframe at the centre.
Network monitoring primarily focused on system performance metrics and error detection within these limited-scale environments. We identified issues manually with basic monitoring tools such as system logs and rudimentary diagnostic utilities.
Client-Server
As my career advanced, I moved into client-server, with organizations deploying networks of interconnected servers and workstations. This architecture created a significant shift in network monitoring as it expanded to include traffic analysis, performance monitoring, and fault management. Simple network management protocol (SNMP) became widely adopted for gathering data from network devices. Administrators could now remotely monitor and manage network elements, paving the way for centralized monitoring solutions.
The Internet Revolution
In the mid-1990s, I moved out of internal support roles and into a client-facing mix of pre- and post-sales support. This coincided nicely with the explosion of the internet and vast interconnected networks. Monitoring solutions had to evolve to handle vast amounts of data and more diverse network topologies. We could monitor network traffic in real-time with network probes and packet sniffers, enabling administrators to identify potential bottlenecks, security threats, and anomalies.
Distributed and Virtualised environments
In the early 2000s, I took a job in the new area of WAN Optimisation, as “employee #1” in the company’s new UK offices, then moving into product management and strategy. This coincided with the development of distributed, virtualised environments, which introduced yet another set of monitoring challenges. With dynamic and elastic infrastructures, traditional monitoring tools struggled. In response, we adopted more sophisticated solutions, such as network flow analysis, which provided insights into traffic patterns, bandwidth utilization, and application behaviour.
Cloud and Software Defined Networking (SDN)
As I moved through product management into strategy, another set of challenges arose as hosting, virtualisation and cloud computing coupled with the emergence of early software-defined WAN (SDWAN) and software-defined networking (SDN) challenged our network monitoring once again. In these environments, monitoring tools had to adapt to infrastructures lacking traditional physical boundaries. SDWAN and SDN allowed administrators to manage and configure network resources dynamically, necessitating monitoring solutions that could adapt to these new environments.
Big Data and Analytics
After a brief period working in Radio Access Networking and IoT, I was lured back to the wonderful world of Observability largely because the rise of big data and analytics had a profound impact on network monitoring. With networks generating enormous volumes of data, monitoring solutions now need to leverage machine learning algorithms and artificial intelligence to identify patterns, detect anomalies, and predict potential issues proactively. Cross-silo data ingestion and real time analytics enable help-desk operators, support engineers and administrators to make fast, accurate, data-driven decisions, enhancing network performance and security.
Security-Centric Monitoring
There is one area in which, believe it or not, I’ve not been directly involved: as cyber threats become more sophisticated and pervasive, network monitoring has evolved to include intrusion detection systems (IDS), security information and event management (SIEM) platforms, and behaviour analytics to detect and mitigate potential security breaches.
It’s never boring
I continue to approach each day working in IT with a sense of wonder and amazement. IT is not “boring” or “run of the mill.” It is constantly changing, evolving, and improving, as we can see from the evolution of network monitoring, which has necessarily mirrored the rapid advancements in computer networking itself.
From basic performance monitoring in early centralized systems to complex analytics-driven Unified Observability solutions in today’s distributed and virtualized networks, network monitoring has become indispensable for ensuring network availability, performance optimization, and robust security. As networks continue to evolve, network monitoring will undoubtedly keep pace, leveraging emerging technologies to meet the ever-growing demands of the digital age.