Sudonix

Sudonix

Technical issue? You might be overthinking it

Blog
  • phenomlabundefined

    Once in every while, you encounter a repetitive issue that no matter what you try to do to resolve it, the problem manifests itself over and over again - sometimes, even on a daily basis. Much of how the issue is remediated really depends on the person assigned to the task.

    You might be puzzled at why I’d write about something like this, but it’s a situation I see constantly - one I like to refer to as “over thinker syndrome”. What do I mean by this ? Here’s the theory. Some people are very analytical when it comes to problem solving. Couple that with technical knowledge and you could land up with a situation where something relatively simple gets blown out of all proportion because the scenario played out in the mind is often much further from reality than you’d expect. And the technical reasoning is usually always to blame. Sometime around 2007, a colleague noticed that the Exchange Server (2003 wouldn’t you know) would suddenly reboot half way through a backup job. Rightly so, he wanted to investigate and asked me if this would be ok. Anyone with an ounce of experience knows that functional backups are critical in the event of a disaster - none more so than I - obviously, I have the go ahead. One bright spark in my team suggested a reboot of the server, which immediately prompted the response

    “…it’s rebooting itself every day, so how will that help ?”

    The investigation

    Joking aside, we’ve all heard the “have you rebooted” question touted at some point during helpdesk discussions, but this one was different. A system rebooting itself is usually symptomatic of an underlying issue somewhere, and my team member was ready for the task ahead. Stepping up to the plate, he asked if it was ok to install some monitoring software on the server. Usually, installing additional software components in a production server without testing first is a non-starter, but seeing as we needed to get this resolved as quickly as possible to reinstate the nightly backup (which incidentally hasn’t run successfully for 3 days by now), I provided approval to proceed without question. There’s a leap of faith at this point, as you could cause more problems than that you actually set out to resolve in the first place, but, as with anything related to information technology, someone’s you have to accept an element of risk. The software itself was actually for the RAID controller and motherboard  The assigned technician had already decided it was related to something along the lines of a faulty RAM module, or perhaps an issue with the controller itself. My thoughts leaned elsewhere already at this point - is the server reboots itself at exactly the same time every day then there is an established pattern which should be investigated first. It’s a logical approach, but it’s a common trait for technical support staff to sometimes think outside of the box - or in this case, outside of the building. Not wanting to push my opinion, or trample on anyone’s toes, I decided to remain quiet and see just how far this would go before intervention was required.

    In this case, not very far. The following morning after another unannounced nightly reboot, the error “the previous shutdown at [insert time and date here] was unexpected” showed up in the event log. No real surprises there, and once again, exactly the same time as the previous night. I asked my technician for an update, and he informed me that he believed that the memory was faulty and somehow causing the server to blue screen and reboot. That was actually a reasonable response and so I commended him on his research and findings, but also reminded him to perform a manual backup so that we at least had something to revert to in the event of a failure. Later that afternoon, the same tech approached me and said that he had ordered some replacement memory, and wanted to arrange downtime to fit it. Trying to keep a poker face and remain passive, I agreed and the memory was replaced the same evening around 10pm. At 2am the following morning, kaboom ! - the server rebooted itself again. Not wanting to admit defeat, our courageous tech suggested that the problem could be due to the system overheating. Another fair point, but not realistic as you’d see this in event log as a thermal shutdown. I willingly entertained this, and allowed investigations into the CPU temperature to begin - after another manual backup. Unsurprisingly, the temperature data returned no smoking gun, so that was abandoned. The next port of call was to reapply the service pack. Now, I’ll admit that this used to fix a multitude of issues under Windows NT Server (particularly Service Pack 4) but not under Windows 2003. I declined this for obvious reasons - if you reapply the service pack, you run the risk of overwriting key DLL files that could (and often will) render Exchange inoperable. Not being prepared to introduce an unprecedented risk into what was already becoming something of a showcase, I suggested that we look elsewhere.

    The exasperation

    The final (and honestly more realistic suggestion) was to enable verbose logging in Exchange. This is actually a good idea, but only if you suspect that the information store could be the issue. Given the evidence, I wasn’t convinced. If there was corruption in the store, or on any of the disks, this would show itself randomly through the day and wouldn’t wait until 2am in the morning. Not wanting to come across as condescending, I agreed, but at the same time, set a deadline to escalation. I wasn’t overly concerned about the backups as these were being completed manually each day whilst the investigations were taking place. Neither was I concerned at what could be seen at this point as wasting someone’s time when you think you may have the answer to what now seemed to be an impossible problem. This is where experience will eclipse any formal qualifications hands down. Those with university degrees may scoff at this, but those with substantially analytical thinking patterns seem to avoid logic like the plague and go off on a wild tangent looking for a dramatically technical explanation and solution to a problem when it’s much simpler than you’d expect. Hence the title of this article - Avoid the “bulldozer to find a china cup” scenario. After witnessing another pained expression on the face of my now exasperated and exhausted tech, I said “let’s get a coffee”. In agreement, he followed me to the kitchen and then asked me what I thought the problem could be. I said that if he wanted my advice, it would be to step back and look at this problem from a logical angle rather than technical. The confused look I received was priceless - the guy must have really though I’d lost the plot. After what seemed like an eternity (although in reality only a few seconds) he asked me what I meant by this. “Come with me”, I said. Finishing his coffee, he diligently followed me to the server room. Once inside, I asked him to show me the Exchange Server. Puzzled, he correctly pointed out the exact machine. I then asked him to trace the power cables and tell me where they went.

    As with most server rooms, locating and identifying cables can be a bit of a challenge after equipment has been added and removed, so this took a little longer than we expected. Eventually, the tech traced the cables back to

    …an old looking UPS that had a red light illuminated at the front like it had been a prop in a Terminator film.

    The realisation

    Suddenly, the real cause of this issue dawned on the tech like a morning sunrise over the Serengeti. The UPS that the Exchange Server was unexpectedly connected to had a faulty battery. The UPS was conducting a self test at 2am each morning, and because the bypass test failed owing to the burnt battery, the connected server lost power and started back up after the offending equipment left bypass mode and went online.

    Where is this going you might ask ?  Here’s the moral of this (particular, and many others like it) story

    • Just because a problem involves technology, it doesn’t mean that the answer has to be a complex technical one
    • Logic and common sense has a part to play in all of our lives.
    • Sometimes, it makes more sense just to step back, take a breath, and see something for what it really is before deciding to commit
    • It’s easy to allow technical expertise to cloud your judgement - don’t fall into the trap of using a sledgehammer to break an egg
    • You cannot buy experience - it’s earned, gained, and leaves an indelible mark

    Let’s hear your views. Did you ever come across a situation where no matter what you tried, nothing worked ? Did the solution turn out to be much simpler than you’d have ever thought ?

    Author of OGProxy, Swatch, Threaded, and most of Sudonix's bugs…

    0
  • phenomlabundefined phenomlab referenced this topic on

  • Madchatthewundefined

    Link vs Refresh

    Solved Customisation
    20
    8 Votes
    20 Posts
    495 Views
    phenomlabundefined

    @pobojmoks Do you see any errors being reported in the console ? At first guess (without seeing the actual code or the site itself), I’d say that this is AJAX callback related

  • phenomlabundefined

    Performing a migration? Ensure you have a back-out plan

    Blog
    1
    +0
    0 Votes
    1 Posts
    169 Views
    phenomlabundefined

    One of the most important safety nets in IT Operations is contingency. Every migration needs a rollback plan in the event that things don’t quite go the way you’d expect, and with a limited timeline to implement a change, or in some cases, a complete migration, the rollback process is one that is an essential component. Without a plan to revert all changes back to their previous state, your migration is destined for failure from the outset. No matter how confident you are (I’ve yet to meet a project manager who doesn’t build in redundancy or rollback in one form or another) there is always going to be something you’ve missed, or a change that produces undesirable results.

    It is this seemingly innocent change that can have a domino effect on your migration - unless you have access to a replica environment, the result of the change cannot be realistically predicted. Admittedly, it’s a simple enough process to clone virtual machines to test against, but that’s of no consequence if your change relates to those conducted at hardware level. A classic example of this is a firewall migration. Whilst it would be possible to test policies to ensure their functionality meets the requirement of the business, confirming VPN links for example isn’t so straightforward - especially when you need to rely on external vendors to complete their piece of the puzzle before you can continue. Unless you’re deploying technology into a greenfield site, you do not have the luxury of testing a VPN into a production network during business hours. Based on this, you have a couple of choices

    You perform all testing off hours by switching equipment for the replacement, and perform end to end testing. Once you are satisfied everything works as it should, you put everything back the way you found it, then schedule a date for the migration. You configure the firewall using a separate subnet, VLAN, and other associated networking elements meaning the two environments run symmetrically

    But which path is the right one ? Good question. There’s no hard and fast rule to which option you go for - although option 2 is more suited to a phased migration approach whilst option 1 is more aligned to “big bang” - in other words, moving everything at the same time. Option 2 is good for testing, but may not reflect reality as you are not targeting the same configuration. As a side note, I’ve often seen situations where residual configuration from option 1 has been left behind, meaning you either land up with a conflict of sorts, or black hole routing.

    Making use of a rollback

    This is where the rollback plan bridges the gap. If you find yourself in a situation where you either run out of time, or cannot continue owing to physical, logical or external constraints, then you would need to invoke your rollback plan. It’s important to note at this stage that part of the project plan should include a point where the progress is reviewed and assessed, and if necessary, the rollback is executed. My personal preference is within around 40% of the allocated time window - all relevant personnel should reconvene and provide status updates around their areas of responsibility, and give a synopsis of any issues - and be fully prepared to elaborate on these if the need arises. If the responsible manager feels that the project is at risk of overrunning it’s started time frame, or cannot be completed within that window, he or she needs to exercise authority to invoke the rollback plan. When setting the review interval, you should also consider the amount of time required to revert all changes and perform regression testing.

    Rollback provides the ideal opportunity to put everything back how it was before you started on your journey - but it does depend on two major factors. Firstly, you need to allocate a suitable time period for the rollback to be completed within. Secondly, unless you have a list of changes that were made to hardware - inclusive of configuration, patching, and a myriad of others, how can you be sure that you’ve covered everything ?

    Time after time I see the same problem - something gets missed, and turns out to be fundamental on Monday morning when the changes haven’t been cross checked.

    So what should a contingency plan consist of ?

    One surefire way to ensure that configurations are preserved prior to making changes is to create backups of running configs - 2 minutes now can save you 2 days of troubleshooting when you can’t remember which change caused your issue.  For virtual machines, this is typically a snapshot that can be restored later should the need arise. A word to the wise though - don’t leave the machine running on snapshot for too​ long as this can rapidly deplete storage space. It’s not a simple process to recover a crashed VM that has run out of disk space.

    Keep version and change control records up to date - particularly during the migration. Any change that could negatively impact the remainder of the project should be examined and evaluated, and if necessary, removed from the scope of works (provided this is a feasible step - sometimes negating a process is enough to make a project fail)

    Document each step. I can’t stress the importance of this enough. I understand that we all want to get things done in a timely manner, but will you realistically remember all the changes you made in the order they were implemented ?
    Use differential tools to examine and easily highlight changes between two configurations. There are a number of free tools on the internet that do this. If you’re using a Windows environment, a personal favourite of mine is WinMerge. Using a diff tool can separate the wood from the trees quickly, and provides a simple overview of changes - very useful in the small hours, I can assure you.
    Working on a switch or firewall ? Learn how to use the CLI. This is often superior in terms of power and usually contains commands that are not available from the GUI. Using this approach, it’s perfectly feasible to bulk load configuration, and also back it out using the same mechanism.

    What if your rollback plan doesn’t work ? Unfortunately, there is absolutely no way to simulate a rollback during project planning, and this is often made worse by many changes being made at once to multiple systems. It’s not that the rollback doesn’t work - it’s usually always a case of settings being reverted before they should be. In most cases, this has the knock on effect of denying yourself access to a system - and it’s always in a place where there are no local support personnel to assist - at least, not immediately. For every migration I have completed over my career, I’ve always ensured that there is an alternative route to reach a remote device should the primary path become inaccessible. For firewalls, this can be a blessing - particularly as they usually permit access on the public interfaces.

    However, delete a route inadvertently and you are toast - you lose access to the firewall full stop - get out of that one. What would I do in a situation like this where the firewall is located in Asia for example, and you are in London ? Again - contingency. You can’t remove a route on a firewall if it was created automatically by the system. In this case, a VLAN or directly connected interface will create it’s own dynamic route, and should still be available. If dealing with a remote firewall, my suggestion here would be Out Of Band Management (OOBM), but not a device connected directly to the firewall itself, as this presents a security risk if not configured properly. A personal preference is a locally connected laptop in the remote location that uses either independent WiFi or a 3G / Mifi presence. Before the migration starts, establish a WebEx or GoToMeeting session (don’t forget to disable UAC here as that can shoot you in the foot), and arrange for a network cable to be plugged into switch fabric, or directly. Direct is better if you can spare the interface, as it removes potential routing issues. Just configure the NIC on the remote machine with an address in the same subnet add the interface you’re connected to, and you’re golden.

    I’ve used the above as a get out of jail free card on several occasions, and I can assure you it works.

    So what are the takeaways here ?

    The most important aspect is to be ready with a response - effectively a “plan b” when things go wrong. Simple planning in advance can save you having to book a flight, or foot the expense of a local IT support firm with no prior knowledge of your network - there’s the security aspect as well; you’d need to provide the password for the device which immediately invokes a change once the remediation is complete. In summary

    Thoroughly plan each migration and allow time for contingency steps. You may not need them, and if you don’t, then you effectively gain time that could be used elsewhere. Have an alternative way of reaching a remote device, and ensure necessary third party vendors are going to be available during your maintenance window should this be necessary. Take regular config backups of all devices. You don’t necessarily need an expensive tool for this - I actually designed a method to make this work using Linux, a TFTP server, and a custom bash script - let me know if you’d like a copy 🙂 Regularly analyse (automated diff) configuration changes between configurations. Any changes that are undocumented or previously approved are a cause for alarm and should be investigated Ensure that you have adequate documentation, and steps necessary to recover systems in the event of failure

    Any thoughts or questions ? Let me know !

  • phenomlabundefined

    How to destroy a community before it's even built

    Blog
    4
    +0
    5 Votes
    4 Posts
    221 Views
    phenomlabundefined

    @crazycells I guess the worst part for me was the trolling - made so much worse by the fact that the moderators allowed it to continue, insisting that the PeerLyst coming was seeing an example by allowing the community to “self moderate” - such a statement being completely ridiculous, and it wasn’t until someone else other than myself pointed out that all of this toxic activity could in fact be crawled by Google, that they decided to step in and start deleting posts.

    In fact, it reached a boiling point where the CEO herself had to step in and post an article stating their justification for “self moderation” which simply doesn’t work.

    The evidence here speaks for itself.

  • phenomlabundefined

    AI... A new dawn, or the demise of humanity ?

    Blog
    74
    +0
    57 Votes
    74 Posts
    2k Views
    phenomlabundefined

    https://news.sky.com/story/ai-poses-growing-threat-to-next-general-election-warns-uk-cyber-security-agency-13007659

  • phenomlabundefined

    We DON'T Need Blinky Lights

    Blog
    1
    0 Votes
    1 Posts
    105 Views
    phenomlabundefined

    1631810017053-netsecurity.jpg.webp
    I read an article By Glenn S. Gerstell (Mr. Gerstell is the general counsel of the National Security Agency) with a great deal of interest. That same article is detailed below

    The National Security Operations Center occupies a large windowless room, bathed in blue light, on the third floor of the National Security Agency’s headquarters outside of Washington. For the past 46 years, around the clock without a single interruption, a team of senior military and intelligence officials has staffed this national security nerve center.

    The center’s senior operations officer is surrounded by glowing high-definition monitors showing information about things like Pentagon computer networks, military and civilian air traffic in the Middle East and video feeds from drones in Afghanistan. The officer is authorized to notify the president any time of the day or night of a critical threat.

    Just down a staircase outside the operations center is the Defense Special Missile and Aeronautics Center, which keeps track of missile and satellite launches by China, North Korea, Russia, Iran and other countries. If North Korea was ever to launch an intercontinental ballistic missile toward Los Angeles, those keeping watch might have half an hour or more between the time of detection to the time the missile would land at the target. At least in theory, that is enough time to alert the operations center two floors above and alert the military to shoot down the missile.

    But these early-warning centers have no ability to issue a warning to the president that would stop a cyberattack that takes down a regional or national power grid or to intercept a hypersonic cruise missile launched from Russia or China. The cyberattack can be detected only upon occurrence, and the hypersonic missile, only seconds or at best minutes before attack. And even if we could detect a missile flying at low altitudes at 20 times the speed of sound, we have no way of stopping it.

    Something I’ve been saying all along is that technology alone cannot stop cyber attacks. Often referred to as a “silver bullet”, or “blinky lights”, this provides the misconception that by purchasing that new, shiny device, you’re completely secure. Sorry folks, but this just isn’t true. In fact, cyber crime, and it’s associated plethora of hourly attacks is evolving at an alarming rate - in fact, much faster than you’d like to believe.

    You’d think that for all the huge technological advances we have made in this world, the almost daily plethora of corporate security breaches, high profile data loss, and individuals being scammed every day would have dropped down to nothing more than a trickle - even to the point where they became virtually non-existent. We are making huge progress with landings on Mars, autonomous space vehicles, artificial intelligence, big data, machine learning, and essentially reaching new heights on a daily basis thanks to some of the most creative minds in this technological sphere. But somehow, we have lost our way, stumbled and fallen - mostly on our own sword. But why ?

    Just like the Y2k Gold Rush in the late 90’s, information security has become the next big thing with companies ranging from a few employees as startups to enterprise organisations touting their services and platforms to be the best in class, and the next “must have” tool in the blue team’s already bulging arsenal of tools. Tools that on their own in fact have little effect unless they are combined with something else as equally as expensive to run. We’ve spent so much time focusing on efforts ranging from what SEIM solution we need to what will be labelled as the ultimate silver bullet capable of eliminating the threat of attack once and for all that in my opinion, we have lost sight of the original goal. With regulatory requirements and best practice pushing us towards products and services that either require additional staff to manage, or are incredibly expensive to deploy and ultimately run. Supposedly, in an effort to simplify the management, analysis, and processing of millions of logs per hour we’ve created even more platforms to ingest this data in order to make sense of it.

    In reality, all we have created is a shark infested pool where larger companies consume up and coming tech startups for breakfast to ensure that they do not pose a threat to their business model / gravy train, therefore enabling them to dominate the space even further with their newly enhanced reach.

    How did we get to this ? What happened to thought process and working together in order to combat the threat that increases on an hourly basis ? We seem to be so focused on making sure that we aren’t the next organisation to be breached that we have lost the art of communication and the full benefit of sharing information so that it assists others in their journey. We’ve become so obsessed with the daily onslaught of platforms that we no longer seem to have the time to even think, let alone take stock and regroup - not as an individual, but as a community.

    There are a number of ”communities” that offer “free” forums and products under the open source banner, but sadly, these seem to be turning into paid-for products at a rate of knots. I understand people need to live and make money, but if awareness was raised to the point where users wouldn’t click links in phishing emails, fall for the fake emergency wire transfer request from the CEO, or be suddenly tempted by the latest offer in terms of cheap technology then we might - just might - be able to make the world a better place. In order to make this work, we first need to remove the stigma that has become so ingrained by the media and set in stone like King Arthur’s Excalibur. Let’s first start with the hacker / criminal parallel. They aren’t the same thing folks.

    Nope. Not at all. Hackers are those people who find ingenious ways of getting into networks and infrastructure that you never even knew existed, trick you into parting with sensitive information (then inform you as to where you went wrong), and most importantly, educate you so that you and your network are far more secure against real attacks and real criminals. These people exist to increase your awareness, and by definition, security footprint - not use it against you in order to steal. Hackers do like to wear hoodies as they are comfortable, but you won’t find one using gloves, wearing a balaclava or sunglasses, and in some cases, they actually prefer desktops rather than laptops.

    The image being portrayed here is one perpetuated by the media, and it has certainly been effective - but not in a positive way. The word “hacker” is now synonymous with criminals, where it really shouldn’t be. One defines security, whereas the other sets out to break it. If we locked up all the hackers on this planet, we’d only have the blue team remaining. It’s the job of the red team (hackers) to see how strong your defences are. Hackers exist to educate, not infiltrate (at least, not without asking for permission first :))

    I personally have lost count of how many times I’ve sat in meetings where a sales pitch around a security platform is touted as a one stop shop or a Swiss army knife that can protect your entire network from a breach. Admittedly, there’s some great technology on the market that performs a variety of functions to protect your estate, but they all fail to take into consideration the weakest link in any chain - users. Irrespective of bleeding edge “combat platforms” (as I like to refer to them), criminals are becoming very adept in their approach, leveraging techniques such as social engineering. It should come as no surprise for you to learn that this type of attack can literally walk past your shiny new defence system as it relies on the one vulnerability you cannot predict - the human. Hence the term “hacking humans”.

    I’m of the firm opinion that if you want to outsmart a criminal, you have to think like one. Whilst newfangled platforms are created to assist in the fight against cyber crime, they are complex to configure, suffer from alerting bloat (far too many emails so you end up missing the one where your network is actually being compromised), or are simply overwhelming and difficult to understand. Here’s the thing. You don’t need (although they do help) expensive bleeding edge platforms with flashing lights to tell you where weak points lie within your network, but you do need to understand how a criminal can and will exploit these. A vulnerability cannot be leveraged if it no longer exists, or even better, never even existed to begin with.

    And so, on with the mission, and the real reason as to why I created this site. I’ve been working in information technology for 30 years, and have a very strong technical background in network design and information security.

    What I want to do is create a communication, information, and awareness sharing platform. I created the original concept of what I thought this new community should look like in my head, but its taken a while to finally develop, get people interested, and on board. To my mind, those from inside and outside of the information security arena will pool together, share knowledge, raise awareness, and probably the most important, harness this new found force and drive change forward.

    The breaches we are witnessing on a daily basis are not going to simply stop. They will increase dramatically in their frequency, and will get worse with each incident.

    Let’s stop the “hackers are criminals” myth, start using our own unique talents in this field, and make a community that

    is able to bring effective change treats everyone as equals The community once fully established could easily be the catalyst for change - both in perception, and inception.

    Why not wield the stick for a change instead of being beaten with it, and work as a global virtual team instead ?

    Will you join me ? In case I haven’t already mentioned it, this initiative has no cost - only gains. It is entirely free.

  • phenomlabundefined

    Security, Or Just Obscurity?

    Blog
    1
    +0
    0 Votes
    1 Posts
    132 Views
    phenomlabundefined

    When you look at your servers or surrounding networks, what exactly do you see ? A work of art, perhaps ? Sadly, this is anything but the picture painted for most networks when you begin to look under the hood. What I’m alluding to here is that beauty isn’t skin deep - in the sense that neat cabling resembling art from the Sistine Chapel, tidy racks, and huge comms rooms full of flashing lights looks appealing from the eye candy perspective and probably will impress clients, but in several cases, this is the ultimate wolf in sheep’s clothing. Sounds harsh ? Of course it does, but with good intentions and reasoning. There’s not a single person responsible for servers and networks on this planet who will willingly admit that whilst his or her network looks like a masterpiece to the untrained eye, it’s a complete disaster in terms of security underneath.

    In reality, it’s quite the opposite. Organisations often purchase bleeding edge infrastructure as a means of leveraging the clear technical advantages, enhanced security, and competitive edge it provides. However, under the impressive start of the art ambience and air conditioning often lies an unwanted beast. This mostly invisible beast lives on low-hanging fruit, will be tempted to help itself at any given opportunity, and is always hungry. For those becoming slightly bewildered at this point, there really isn’t an invisible beast lurking around your network that eats fruit. But, with a poorly secured infrastructure, there might as well be. The beast being referenced here is an uninvited intruder in your network. A bad actor, threat actor, bad guy, criminal…. call it what you want (just don’t use the word hacker) can find their way inside your network by leveraging the one thing that I’ve seen time and time again in literally every organisation I ever worked for throughout my career - the default username and password. I really can’t stress the importance of changing this on new (and existing) network equipment enough, and it doesn’t stop at this either.

    Changing the default username and password is about 10% of the puzzle when it comes to security and basic protection principles. Even the most complex credentials can be bypassed completely by a vulnerability (or in some cases, a backdoor) in ageing firmware on switches, firewalls, routers, storage arrays, and a wealth of others - including printers (which incidentally make an ideal watering hole thanks to the defaults of FTP, HTTP, SNMP, and Telnet, most (if not all of) are usually always on. As cheaper printers do not have screens like their more expensive copier counterparts (the estate is reduced to make the device smaller and cheaper), any potential criminal can hide here and not be detected - often for months at a time - arguably, they could live in a copier without you being aware also. A classic example of an unknown exploit into a system was the Juniper firewall backdoor that permitted full admin access using a specific password - regardless of the one set by the owner. Whilst the Juniper exploit didn’t exactly involve a default username and password as such (although this particular exploit was hard-coded into the firmware, meaning that any “user” with the right coded password and SSH access remotely would achieve full control over the device), it did leverage the specific vulnerability in the fact that poorly configured devices could have SSH configured as accessible to 0.0.0.0/0 (essentially, the entire planet) rather than a trusted set of IP addresses - typically from an approved management network.

    We all need to get out of the mindset of taking something out of a box, plugging it into our network, and then doing nothing else - such as changing the default username and password (ideally disabling it completely and replacing it with a unique ID) or turning off access protocols that we do not want or need. The real issue here is that today’s technology standards make it simple for literally anyone to purchase something and set it up within a few minutes without considering that a simple port scan of a subnet can reveal a wealth of information to an attacker - several of these tools are equipped with a default username and password dictionary that is leveraged against the device in question if it responds to a request. Changing the default configuration instead of leaving it to chance can dramatically reduce the attack landscape in your network. Failure to do so changes “plug and play” to “ripe for picking”, and its those devices that perform seemingly “minor” functions in your network that are the easiest to exploit - and leverage in order to gain access to neighbouring ancillary services. Whilst not an immediate gateway into another device, the compromised system can easily give an attacker a good overview of what else is on the same subnet, for example.

    So how did we arrive at the low hanging fruit paradigm ?

    It’s simple enough if you consider the way that fruit can weigh down the branch to the point where it is low enough to be picked easily. A poorly secured network contains many vulnerabilities that can be leveraged and exploited very easily without the need for much effort on the part of an attacker. It’s almost like a horse grazing in a field next to an orchard where the apples hang over the fence. It’s easily picked, often overlooked, and gone in seconds. When this term is used in information security, a common parallel is the path of least resistance. For example, a pickpocket can acquire your wallet without you even being aware, and this requires a high degree of skill in order to evade detection yet still achieve the primary objective. On the other hand, someone strolling down the street with an expensive camera hanging over their shoulder is a classic example of the low hanging fruit synopsis in the sense that this theft is based on an opportunity that wouldn’t require much effort - yet with a high yield. Here’s an example of how that very scenario could well play out.

    Now, as much as we’d all like to handle cyber crime in this way, we can’t. It’s illegal 🙂

    What isn’t illegal is prevention. 80% of security is based on best practice. Admittedly, there is a fair argument as to what exactly is classed as “best” these days, although it’s a relatively well known fact that patching the Windows operating system for example is one of the best ways to stamp out a vulnerability - but only for that system that it is designed to protect against. Windows is just the tip of the iceberg when it comes to vulnerabilities - it’s not just operating systems that suffer, but applications, too. You could take a Windows based IIS server, harden it in terms of permitted protocols and services, plus install all of the available patches - yet have an outdated version of WordPress running (see here for some tips on how to reduce that threat), or often even worse, outdated plugins that allow remote code execution. The low hanging fruit problem becomes even more obvious when you consider breaches such as the well-publicised Mossack Fonseca (“Panama Papers”). What became clear after an investigation is that the attackers in this case leveraged vulnerabilities in the firm’s WordPress and Joomla public facing installations - this in fact led to them being able to exploit an equally vulnerable mail server by brute-forcing it.

    So what should you do ? The answer is simple. It’s harvest time.

    If there is no low-hanging fruit to pick, life becomes that much more difficult for any attacker looking for a quick “win”. Unless determined, it’s unlikely that your average attacker is going to spend a significant amount of time on a target (unless it’s Fort Knox - then you’ve have to question the sophistication) then walk away empty handed with nothing to show for the effort. To this end, below are my top recommendations. They are not new, non-exhaustive, and certainly not rocket science - yet they are surprisingly missing from the “security 101” perspective in several organisations.

    Change the default username and password on ALL infrastructure. It doesn’t matter if it’s not publicly accessible - this is irrelevant when you consider the level of threats that have their origins from the inside. If you do have to keep the default username (in other words, it can’t be disabled), set the lowest possible access permissions, and configure a strong password. Close all windows - in this case, lock down protocols and ports that are not essential - and if you really do need them open, ensure that they are restricted Deploy MFA (or at least 2FA) to all public facing systems and those that contain sensitive or personally identifiable information Deploy adequate monitoring and logging techniques, using a sane level of retention. Without any way of forensic examination, any bad actor can be in and out of your network well before you even realise a breach may have taken place. The only real difference is that without decent logging, you have no way of confirming or even worse, quantifying your suspicion. This can spell disaster in regulated industries. Don’t shoot yourself in the foot. Ensure all Windows servers and PC’s are up to date with the latest patches. The same applies to Linux and MAC systems - despite the hype, they are vulnerable to an extent (but not in the same way as Windows), although attacks are notoriously more difficult to deploy and would need to be in the form of a rootkit to work properly Do not let routers, firewalls, switches, etc “slip” in terms of firmware updates. Keep yourself and your team regularly informed and updated around the latest vulnerabilities, assess their impact, and most importantly, plan an update review. Not upgrading firmware on critical infrastructure can have a dramatic effect on your overall security. Lock down USB ports, CD/DVD drives, and do not permit access to file sharing, social media, or web based email. This has been an industry standard for years, but you’d be surprised at just how many organisations still have these open and yet, do not consider this a risk. Reduce the attack vector by segmenting your network using VLANS. For example, the sales VLAN does not need to (and shouldn’t need to) connect directly to accounting etc. In this case, a ransomware or malware outbreak in sales would not traverse to other VLANS, therefore, restricting the spread. A flat network is simple to manage, but a level playing field for an attacker to compromise if all the assets are in the same space. Don’t use an account with admin rights to perform your daily duties. There’s no prizes for guessing the level of potential damage this can cause if your account is compromised, or you land up with malware on your PC Educate and phish your users on a continual basis. They are the gateway directly into your network, and bypassing them is surprisingly easy. You only have to look at the success of phishing campaigns to realise that they are (and always will be) the weakest link in your network. Devise a consistent security and risk review framework. Conducting periodic security reviews is always a good move, and you’d be surprised at just what is lurking around on your network without your knowledge. There needn’t be a huge budget for this. There are a number of open source projects and platforms that make this process simple in terms of identification, but you’ll still need to complete the “grunt” work in terms of remediation. I am currently authoring a framework that will be open source, and will share this with the community once it is completed. Conduct regular governance and due diligence on vendors - particularly those that handle information considered sensitive (think GDPR). If their network is breached, any information they hold around your network and associated users is also at risk. Identify weak or potential risk areas within your network. Engage with business leaders and management to raise awareness around best practice, and information security. Perform breach simulation, and engage senior management in this exercise. As they are the fundamental core of the business function, they also need to understand the risk, and more importantly, the decisions and communication that is inevitable post breach.

    There is no silver bullet when it comes to protecting your network, information, and reputation. However, the list above will form the basis of a solid framework.

    Let’s not be complacent - let’s be compliant.

  • phenomlabundefined

    Keep It Simple, Stupid...

    Blog
    1
    0 Votes
    1 Posts
    298 Views
    phenomlabundefined

    tech.jpeg
    Ever heard of KISS ? Nope - not these guys

    kiss.jpeg
    What I’m referring to is the acronym was reportedly coined by Kelly Johnson, lead engineer at the Lockheed Skunk Works (creators of the Lockheed U-2 and SR-71 Blackbird spy planes, among many others), which formed the basis of the relationship between the way things break, and the sophistication available to repair them. You might be puzzled at why I’d write about something like this, but it’s a situation I see constantly – one I like to refer to as “over thinker syndrome”. What do I mean by this ? Here’s the theory. Some people are very analytical when it comes to problem solving. Couple that with technical knowledge and you could land up with a situation where something relatively simple gets blown out of all proportion because the scenario played out in the mind is often much further from reality than you’d expect. And the technical reasoning is usually always to blame.

    Some years ago in a previous career, a colleague noticed that the Exchange Server (2003 wouldn’t you know) would suddenly reboot half way through a backup job. Rightly so, he wanted to investigate and asked me if this would be ok. Anyone with an ounce of experience knows that functional backups are critical in the event of a disaster – none more so than I – obviously, I gave the go ahead. One bright spark in my team suggested a reboot of the server, which immediately prompted the response

    “……it’s rebooting itself every day, so how will that help ?”

    There’s always one, isn’t there ? The final (and honestly more realistic suggestion) was to enable verbose logging in Exchange. This is actually a good idea, but only if you suspect that the information store could be the issue. Given the evidence, I wasn’t convinced. If there was corruption in the store, or on any of the disks, this would show itself randomly through the day and wouldn’t wait until 2am in the morning. Not wanting to come across as condescending, I agreed, but at the same time, set a deadline to escalation. I wasn’t overly concerned about the backups as these were being completed manually each day whilst the investigations were taking place. Neither was I concerned at what could be seen at this point as wasting someone’s time when you think you may have the answer to what now seemed to be an impossible problem. This is where experience will eclipse any formal qualifications hands down. Those with university degrees may scoff at this, but those with substantially analytical thinking patterns seem to avoid logic like the plague and go off on a wild tangent looking for a dramatically technical explanation and solution to a problem when it’s much simpler than you’d expect.

    After witnessing the pained expression on the face of my now exasperated and exhausted tech, I said “let’s get a coffee”. In agreement, he followed me to the kitchen and then asked me what I thought the problem could be. I said that if he wanted my advice, it would be to step back and look at this problem from a logical angle rather than technical. The confused look I received was priceless – the guy must have really though I’d lost the plot. After what seemed like an eternity (although in reality only a few seconds) he asked me what I meant by this. “Come with me”, I said. Finishing his coffee, he diligently followed me to the server room. Once inside, I asked him to show me the Exchange Server. Puzzled, he correctly pointed out the exact machine. I then asked him to trace the power cables and tell me where they went.

    As with most server rooms, locating and identifying cables can be a bit of a challenge after equipment has been added and removed, so this took a little longer than we expected. Eventually, the tech traced the cables back to

    ………an old looking UPS that had a red light illuminated at the front like it had been a prop in a Terminator film.

    Suddenly, the real cause of this issue dawned on the tech like a morning sunrise over the Serengeti. The UPS that the Exchange Server was unexpectedly connected to had a faulty battery. The UPS was conducting a self test at 2am each morning, and because the bypass test failed owing to the burnt battery, the connected server lost power and started back up after the offending equipment left bypass mode and went online.

    Where is this going you might ask ? Here’s the moral of this particular story

    Just because a problem involves technology, it doesn’t mean that the answer has to be a complex technical one Logic and common sense has a part to play in all of our lives. Sometimes, it makes more sense just to step back, take a breath, and see something for what it really is before deciding to commit It’s easy to allow technical expertise to cloud your judgement – don’t fall into the trap of using a sledgehammer to break an egg You cannot buy experience – it’s earned, gained, and leaves an indelible mark
  • phenomlabundefined

    RESOLVED: Blog offline for maintenance

    Announcements
    9
    3 Votes
    9 Posts
    274 Views
    gotwfundefined

    Well, just remember - No matter where ya’ go, there you are. 🏇 🐎 🐴