Neural networks being used to create realistic phishing emails

Security
  • It would appear that there are ever increasing instances where AI-empowered chatbots and neural networks such as OpenAI’s ChatGPT have been used to create phishing emails that evade standard security detections due to the lack of typical spelling, grammar, and syntax errors that are commonly found in such emails.

    https://openai.com/blog/chatgpt/

    These chatbots are also capable of supplying content for misinformation and disinformation campaigns given their advanced writing capabilities that allow the generation of entire documents and forum / social media posts with both persuasive language and speed. Previously, spotting a poorly constructed phishing email was a relatively simple exercise owing to obvious spelling and grammatical mistakes, but this is slowly becoming a thing of the past owing to the rise of AI powered chatbots.

    You’ve likely encountered chatbots when asking for support on a retail site, or with your online bank – these seemingly “helpful” (sometimes 😊) attendants are based on machine learning, and can quickly adapt a conversation based on input from the requester. Whilst some of these chatbots are still very synthetic in nature, ChatGPT is an advanced system that can very easily make it appear you are talking to another human. See enclosed for an example – in this case, it’s even smart enough to question the ethics of a discussion before it continues after receiving validation that the user intends to secure their own property, and not break into someone else’s.

    During its learning and training phase, ChatGPT is actually free to use and try out. This has the unfortunate side effect of making it an invaluable tool for cyber criminals who are currently leveraging it’s capabilities in order to evade detection from traditional rulesets designed to stop email based on grammar and other authoring techniques. Previous campaigns often used “keyword stuffing” which is a technique designed to confuse older protection models by inserting random words in other existing text making them nonsensical, but allowing them to bypass older and less reliable filters because the standard checking algorithms are unable to determine if they are fake or not.

    ChatGPT has also been used in some nefarious campaigns to make it look like you are conversing with a human, when in fact, it is under the control of a malicious actor with criminal intent. This relatively new technology inevitably opens the floodgates for cyber criminals, and due to it’s convincing nature, it can easily make malicious emails appear harmless in nature, look legitimate, and therefore increasing the successful delivery rate of such content.

  • Here’s the image I referenced in the first post

    943c0dc0-417d-4080-a02a-56ebf5592c56-image.png

    You can also see it in action for yourself here (form an orderly queue, it’s very popular). You can however keep pressing F5 to see what the bot is being asked to do.

    https://chat.openai.com/auth/login

    270e3b3a-f6c6-4f80-bbc1-cf3598775d44-image.png

  • phenomlabundefined phenomlab marked this topic as a regular topic on
  • @phenomlab I think they have updated ChatGPT recently. It limits the number of questions you can ask, and additionally does not answer “illegal” questions the same way anymore…

    And last week, I saw someone on reddit, that was blocked by ChatGPT because he was asking something illegal on chat.

    It will be interesting to see how it will evolve. This evolution will probably help all other companies (like Google) that are exploring this field…

  • @crazycells Good to see that it is able to spot nefarious attempts in an effort to exploit it. I must admit, I’m no fan of AI or ML and wrote an article about that here. I’ve tried to make this balanced, and not all “Hollywood” 🙂

    https://sudonix.org/topic/138/ai-a-new-dawn-or-the-demise-of-humanity

  • phenomlabundefined phenomlab referenced this topic on
  • after several months of abuse by the internet community 🙂 this is what ChatGPT became…

    alt text

  • @phenomlab recently, I have observed something interesting…

    When I asked chatgpt for a recently published work about two scientific fields (in this case, epigenetics and aging) , it gave me an answer and list some article names.

    At first, I was shocked to miss these papers, and not seeing them before… but later I figured out these are not real papers. 🙂 Although, I have to admit, it sounds and looks legit, it is definitely how those article titles are constructed, even some of the authors are real scientists who are working in these fields… But I could not find any of these publications, so they do not exist 😄

    Somehow, chatgpt understands and learns how to give these answers, but it could not make the connection that these references should be real and represent something that exists…

    Screen Shot 2023-01-20 at 18.26.53.png

  • @crazycells very interesting indeed. Particularly to provide works that do not exist, and yet reference known professionals in those fields as the authors when they are not.

  • @phenomlab said in Neural networks being used to create realistic phishing emails:

    @crazycells very interesting indeed. Particularly to provide works that do not exist, and yet reference known professionals in those fields as the authors when they are not.

    yes, actually I have to add something…

    I only identified two people, the rest of the authors did not exist. But one of those two people is a very well-known person in the field that has a lot of review articles… so that might be the reason… chatgpt assumed his name should appear on the list 🙂

  • @phenomlab you may remember that people were writing a lot of assays or articles using chatgpt in the first week and many people were discussing how a lot of students will use this in their coursework.

    I heard that a teacher was able to identify the homework assays written by chatgpt by directly asking chatgpt if it wrote or not 🙂 lol , I did not read this but heard from a friend. that is funny, so chatgpt is a snitch

    snitch

    And I have read a few days ago, chatgpt makes the assays by known/predictable structures (how many sentences in a paragraph, how many words in a sentence, what kind of words should be used etc.) I guess that is the average of the many assays it scanned… that is why the assays look legit…

  • @crazycells chatGPT is a snitch… Love it.

    On a serious note, you can’t help but wonder if the same algorithm is being used by teachers who use AI to detect plagiarism in essays to eliminate cheating.

  • @phenomlab yes, I believe chatgpt (and its competitors) will be the first step of action for a lot of people especially young people soon 😄

    I have just asked for a job application email sample and a CV sample separately… it gave me very good samples… Of course, it is nowhere near polished or finished , but as a “first step” , I think it is very acceptable. Many people can utilize it especially if you are not professional. I do not think it is as sophisticated yet for professionals, but will become there at some point I guess…

    you can even make it write a code for you, lol… I have just asked this:

    Screen Shot 2023-01-20 at 21.15.12.png

  • @crazycells I bet the CV from chatGPT is of a higher quality than some I’ve received from real people in the past. Seriously, they are so bad in terms of basic spelling and grammar - which is inexcusable given that you have a spell and grammar checker right in front of you - that they often don’t make it out of the gate, and to interview stage.

    Obviously, I make allowances for English not being the primary language or “mother tongue” when accepting CV’s from potential candidates overseas (and I do the same here - I won’t correct anything anyone posts on sudonix provided it remains within the guidelines), but these are from UK residents which makes it so much worse. The point here is that you could use something like chatGPT to compose your CV, but you’d be very quickly exposed if your grammar and spelling weren’t up to scratch 😧

  • @phenomlab said in Neural networks being used to create realistic phishing emails:

    @crazycells I bet the CV from chatGPT is of a higher quality than some I’ve received from real people in the past. Seriously, they are so bad in terms of basic spelling and grammar - which is inexcusable given that you have a spell and grammar checker right in front of you - that they often don’t make it out of the gate, and to interview stage.

    Obviously, I make allowances for English not being the primary language or “mother tongue” when accepting CV’s from potential candidates overseas (and I do the same here - I won’t correct anything anyone posts on sudonix provided it remains within the guidelines), but these are from UK residents which makes it so much worse. The point here is that you could use something like chatGPT to compose your CV, but you’d be very quickly exposed if your grammar and spelling weren’t up to scratch 😧

    yes, CV sample is quite acceptable as a first draft… I agree it would eliminate all the small mistakes. I will try to use it more commonly and figure out more ways to get benefit from it 😄

  • Just came across this which made me laugh. Very much in keeping with this topic

    image_9a734d4b-3e80-4ae0-ae86-ff64bb10f63f20230124_155314.jpg

  • @phenomlab lol, good… chatgpt is adapting into human culture…

  • @crazycells just came across this. Looks like Google has finally jumped into the bandwagon with it’s own offering called “Bard”

    https://news.sky.com/story/google-launches-ai-chatbot-bard-to-rival-wildly-successful-chatgpt-12804958

  • @phenomlab so here it begins… AI wars… we have to protect John Connor no matter what happens…

  • @crazycells wondered how long it would be before the Hollywood connotation got a mention 😁. In all seriousness, it’ll be interesting to see how this inevitable battle of the giants will play out.

  • @phenomlab said in Neural networks being used to create realistic phishing emails:

    @crazycells wondered how long it would be before the Hollywood connotation got a mention

    lol…

    what if Bard starts chatting with ChatGPT and they realize that Homo Sapiens is inferior to them, so they join forces to form SkyNet to enslave us ? 😆


  • 16 Votes
    21 Posts
    161 Views

    @crazycells said in How long before AI takes over your job?:

    sponsored content

    To me, this is the method to get yourself to the top of the list. Unfair advantage doesn’t even properly describe it.

  • 2 Votes
    12 Posts
    60 Views

    @DownPW looks good to me.

  • 11 Votes
    47 Posts
    275 Views

    @DownPW Seems fine.

  • 12 Votes
    17 Posts
    67 Views

    @phenomlab lol yeap, very smart… I read it and immediately ask the same question to ChatGPT and saved the letter sample 😄

    I might use it in the future.

  • 2 Votes
    8 Posts
    255 Views

    @DownPW great response - and salent points well made.

  • 5 Votes
    29 Posts
    532 Views

    @phenomlab said in nodebb chat roll dice game:

    @DownPW I still think you could do something much quicker with jQuery.

    Why not but like I said, I have no skills to do that.

    If you are motivated, why not but I don’t want to bother you especially since it will only be for a certain period of time.

  • 3 Votes
    3 Posts
    259 Views

    @downpw Yes, exactly. Sudonix is about much more than NodeBB 🙂

  • 0 Votes
    1 Posts
    298 Views

    tech.jpeg
    Ever heard of KISS ? Nope - not these guys

    kiss.jpeg
    What I’m referring to is the acronym was reportedly coined by Kelly Johnson, lead engineer at the Lockheed Skunk Works (creators of the Lockheed U-2 and SR-71 Blackbird spy planes, among many others), which formed the basis of the relationship between the way things break, and the sophistication available to repair them. You might be puzzled at why I’d write about something like this, but it’s a situation I see constantly – one I like to refer to as “over thinker syndrome”. What do I mean by this ? Here’s the theory. Some people are very analytical when it comes to problem solving. Couple that with technical knowledge and you could land up with a situation where something relatively simple gets blown out of all proportion because the scenario played out in the mind is often much further from reality than you’d expect. And the technical reasoning is usually always to blame.

    Some years ago in a previous career, a colleague noticed that the Exchange Server (2003 wouldn’t you know) would suddenly reboot half way through a backup job. Rightly so, he wanted to investigate and asked me if this would be ok. Anyone with an ounce of experience knows that functional backups are critical in the event of a disaster – none more so than I – obviously, I gave the go ahead. One bright spark in my team suggested a reboot of the server, which immediately prompted the response

    “……it’s rebooting itself every day, so how will that help ?”

    There’s always one, isn’t there ? The final (and honestly more realistic suggestion) was to enable verbose logging in Exchange. This is actually a good idea, but only if you suspect that the information store could be the issue. Given the evidence, I wasn’t convinced. If there was corruption in the store, or on any of the disks, this would show itself randomly through the day and wouldn’t wait until 2am in the morning. Not wanting to come across as condescending, I agreed, but at the same time, set a deadline to escalation. I wasn’t overly concerned about the backups as these were being completed manually each day whilst the investigations were taking place. Neither was I concerned at what could be seen at this point as wasting someone’s time when you think you may have the answer to what now seemed to be an impossible problem. This is where experience will eclipse any formal qualifications hands down. Those with university degrees may scoff at this, but those with substantially analytical thinking patterns seem to avoid logic like the plague and go off on a wild tangent looking for a dramatically technical explanation and solution to a problem when it’s much simpler than you’d expect.

    After witnessing the pained expression on the face of my now exasperated and exhausted tech, I said “let’s get a coffee”. In agreement, he followed me to the kitchen and then asked me what I thought the problem could be. I said that if he wanted my advice, it would be to step back and look at this problem from a logical angle rather than technical. The confused look I received was priceless – the guy must have really though I’d lost the plot. After what seemed like an eternity (although in reality only a few seconds) he asked me what I meant by this. “Come with me”, I said. Finishing his coffee, he diligently followed me to the server room. Once inside, I asked him to show me the Exchange Server. Puzzled, he correctly pointed out the exact machine. I then asked him to trace the power cables and tell me where they went.

    As with most server rooms, locating and identifying cables can be a bit of a challenge after equipment has been added and removed, so this took a little longer than we expected. Eventually, the tech traced the cables back to

    ………an old looking UPS that had a red light illuminated at the front like it had been a prop in a Terminator film.

    Suddenly, the real cause of this issue dawned on the tech like a morning sunrise over the Serengeti. The UPS that the Exchange Server was unexpectedly connected to had a faulty battery. The UPS was conducting a self test at 2am each morning, and because the bypass test failed owing to the burnt battery, the connected server lost power and started back up after the offending equipment left bypass mode and went online.

    Where is this going you might ask ? Here’s the moral of this particular story

    Just because a problem involves technology, it doesn’t mean that the answer has to be a complex technical one Logic and common sense has a part to play in all of our lives. Sometimes, it makes more sense just to step back, take a breath, and see something for what it really is before deciding to commit It’s easy to allow technical expertise to cloud your judgement – don’t fall into the trap of using a sledgehammer to break an egg You cannot buy experience – it’s earned, gained, and leaves an indelible mark