Skip to content
Sudonix

Export posts out of NodeBB into HTML and Markdown flat files -> Halo ITSM

Guides
1 1 575 1
  • phenomlabundefined

    At work, we are transitioning from NodeBB for our Knowledge Base to Halo ITSM, which we require for SOC2 compliance amongst other things. Because I had 165 articles in NodeBB I didn’t want to have to re-type, or even copy and paste, I decided to write a Python script to walk the target category and create a file for each.

    Here’s the script to complete that. There are a number of prerequisities here, which I’ve identified below

    import os
import re
import time
import requests
import html2text
from datetime import datetime

# --- CONFIGURATION ---
# Your Forum URL goes here
BASE_URL = "https:/yourforum.com"
#The category ID you want to target goes here
CATEGORY_ID = 3
# In my case, I needed to define a new "home" for the exported files under `/public/uploads` as this contained all the images I needed to embed into the new flat files. Therefore, ASSET_DOMAIN is nothing more than a basic website where I can grab the images from afterwards.
ASSET_DOMAIN = "https://assetlocation.com"
# The below directories are created at the same level as the script. If they do not exist, you need to create them. They will contain both `HTML`  and `markdown` copies of the posts.
HTML_DIR = "nodebb_export_html"
MD_DIR = "nodebb_export_markdown"
os.makedirs(HTML_DIR, exist_ok=True)
os.makedirs(MD_DIR, exist_ok=True)

h = html2text.HTML2Text()
h.ignore_links = False
h.body_width = 0

page = 1
total_exported = 0

print(f"🔄 Starting export for category {CATEGORY_ID} from {BASE_URL}")

while True:
    print(f"📄 Fetching page {page}...")
    url = f"{BASE_URL}/api/category/{CATEGORY_ID}?page={page}"
    res = requests.get(url, timeout=10)
    if res.status_code != 200:
        print(f"❌ Failed to fetch page {page}: {res.status_code}")
        break

    data = res.json()
    topics = data.get("topics", [])
    if not topics:
        print("✅ No more topics found. Export complete.")
        break

    for topic in topics:
        tid = topic['tid']
        title = topic['title']
        print(f"→ Exporting topic {tid}: {title}")

        topic_url = f"{BASE_URL}/api/topic/{tid}"
        topic_res = requests.get(topic_url, timeout=10)
        if topic_res.status_code != 200:
            print(f"⚠️ Failed to fetch topic {tid}")
            continue

        topic_data = topic_res.json()
        posts = topic_data.get("posts", [])
        tags = topic_data.get("topic", {}).get("tags", [])
        tag_list = ", ".join(tags) if tags else ""

        safe_title = title.replace(' ', '_').replace('/', '-')
        html_file = f"{HTML_DIR}/{tid}-{safe_title}.html"
        md_file = f"{MD_DIR}/{tid}-{safe_title}.md"

        # --- HTML EXPORT ---
        with open(html_file, "w", encoding="utf-8") as f_html:
            f_html.write(f"<html><head><title>{title}</title></head><body>\n")
            f_html.write(f"<h1>{title}</h1>\n")
            if tag_list:
                f_html.write(f"<p><strong>Tags:</strong> {tag_list}</p>\n")

            for post in posts:
                username = post['user']['username']
                content_html = post['content']
                timestamp = datetime.utcfromtimestamp(post['timestamp'] / 1000).strftime('%Y-%m-%d %H:%M:%S UTC')
                pid = post['pid']

                # Rewrite asset paths in HTML
                content_html = re.sub(
                    r'src=["\'](/assets/uploads/files/.*?)["\']',
                    rf'src="{ASSET_DOMAIN}\1"',
                    content_html
                )
                content_html = re.sub(
                    r'href=["\'](/assets/uploads/files/.*?)["\']',
                    rf'href="{ASSET_DOMAIN}\1"',
                    content_html
                )

                f_html.write(f"<div class='post'>\n")
                f_html.write(f"<h3><strong>Original Author: {username}</strong></h3>\n")
                f_html.write(f"<p><em>Posted on: {timestamp} &nbsp;|&nbsp; Post ID: {pid}</em></p>\n")
                f_html.write(f"{content_html}\n")
                f_html.write("<hr/>\n</div>\n")

            f_html.write("</body></html>\n")

        # --- MARKDOWN EXPORT ---
        with open(md_file, "w", encoding="utf-8") as f_md:
            # Metadata block
            f_md.write(f"<!-- FAQLists: Knowledge Base -->\n")
            if tag_list:
                f_md.write(f"<!-- Tags: {tag_list} -->\n")
            f_md.write("\n")

            f_md.write(f"# {title}\n\n")

            for post in posts:
                username = post['user']['username']
                content_html = post['content']
                timestamp = datetime.utcfromtimestamp(post['timestamp'] / 1000).strftime('%Y-%m-%d %H:%M:%S UTC')
                pid = post['pid']

                # Convert HTML to Markdown
                content_md = h.handle(content_html).strip()

                # Rewrite asset paths
                content_md = re.sub(
                    r'(!\[.*?\])\((/assets/uploads/files/.*?)\)',
                    rf'\1({ASSET_DOMAIN}\2)',
                    content_md
                )
                content_md = re.sub(
                    r'(\[.*?\])\((/assets/uploads/files/.*?)\)',
                    rf'\1({ASSET_DOMAIN}\2)',
                    content_md
                )

                f_md.write(f"**Original Author: {username}**\n\n")
                f_md.write(f"_Posted on: {timestamp}  |  Post ID: {pid}_\n\n")
                f_md.write(f"{content_md}\n\n---\n\n")

        total_exported += 1
        print(f"✔ Saved: {html_file} & {md_file}")

    page += 1
    time.sleep(1)

print(f"\n🎉 Done! Exported {total_exported} topics to '{HTML_DIR}' and '{MD_DIR}'")

    Run the script using python scriptname.py.

    If the script fails, it’s likely because you do not have the required modules installed in Python

    import os
import re
import time
import requests
import html2text

    In this case, you’d need to install them using (for example) pip install html2text

    To get them into an Excel file where they can all be bulk imported, we’d then use something like the below script

    import os
import re
import pandas as pd
from datetime import datetime
import markdown

# --- CONFIGURATION ---
export_dir = "nodebb_export_markdown"
output_file = "Halo_KB_Import_HTML.xlsx"
# This value can be whatever suits your needs
created_by = "Import"
today = datetime.today().strftime('%Y-%m-%d')

# --- BUILD DATAFRAME FOR HALO ---
import_rows = []

for filename in sorted(os.listdir(export_dir)):
    if filename.endswith(".md"):
        filepath = os.path.join(export_dir, filename)
        with open(filepath, "r", encoding="utf-8") as f:
            lines = f.readlines()

        # Default values
# Change "Knowledge Base" to reflect what you are using in Halo
        faqlists = "Knowledge Base"
        tags = ""

        # Parse metadata comments from top of file
        metadata_lines = []
        while lines and lines[0].startswith("<!--"):
            metadata_lines.append(lines.pop(0).strip())

        for line in metadata_lines:
            faq_match = re.match(r"<!-- FAQLists:\s*(.*?)\s*-->", line)
            tag_match = re.match(r"<!-- Tags:\s*(.*?)\s*-->", line)

            if faq_match:
                faqlists = faq_match.group(1)
            if tag_match:
                tags = tag_match.group(1)

        markdown_content = ''.join(lines)
        html_content = markdown.markdown(markdown_content)

        # Extract summary from filename
        summary = filename.split('-', 1)[1].rsplit('.md', 1)[0].replace('_', ' ')

        import_rows.append({
            "Summary": summary,
            "Details": html_content,
            "Resolution": "",
            "DateAdded": today,
            "CreatedBy": created_by,
            "FAQLists": faqlists,
            "Tags": tags
        })

# --- EXPORT TO EXCEL ---
df = pd.DataFrame(import_rows)
df.to_excel(output_file, index=False)

print(f"✅ Done! Halo HTML import file created: {output_file}")

    This then generates a file called Halo_KB_Import_HTML.xlsx which you can then use to import each exported post into Halo.

    Cool eh? Huge time saver 🙂

    Mark – Founder, Phenomlab Ltd
    Executive IT & Security Leadership
    Phenomlab Ltd

    3
Log in to reply

Related Topics
  • cagatayundefined

    NodeBB v4.0.0

    General nodebb sudonix version 4
    28
    4 Votes
    28 Posts
    4k Views
    phenomlabundefined
    @Panda said in NodeBB v4.0.0: the workings of World aren’t intuitive Its not easy to get World populating when a forum is new to it This is a good point and one I’ve considered also. It’s a little confusing to be honest.
  • crazycellsundefined

    CSS code customization for the link preview plugin

    Solved Customisation css nodebb
    6
    2
    3 Votes
    6 Posts
    2k Views
    phenomlabundefined
    @kadir-ay-0 marking as resolved based on https://community.nodebb.org/topic/17109/manual-build-a-night-mode-for-harmony/5 Please do not raise requests in two places - here and the NodeBB forums. All this does is create unnecessary load for both parties.
  • crazycellsundefined

    navigation menu panel on mobile

    Solved Customisation nodebb css
    8
    1
    7 Votes
    8 Posts
    2k Views
    phenomlabundefined
    @crazycells hmm. That’s odd. I haven’t made any changes from recollection but I could be wrong. I’ll need to check. EDIT - very strange. I honestly don’t recall adding the below CSS block to alter the bottom bar, but you’re right… .bottombar-nav { padding: 0px !important; } I’ve removed this so it reflects stock Harmony.
  • dave1904undefined

    Plugin to show images in teasers

    General nodebb teaser image plugin hook
    6
    1
    2 Votes
    6 Posts
    1k Views
    phenomlabundefined
    @dave1904 I’d start by adding a console.log function to hookData so you can see what is being returned return hookData; console.log(hookData):
  • cagatayundefined

    Top Ranked Forums

    Chitchat nodebb top ranked
    9
    1
    3 Votes
    9 Posts
    2k Views
    phenomlabundefined
    The real issue here is that most people consider forums to be “dead” in the sense that nobody uses them anymore, and social media groups have taken their place. Their once dominant stance in the 90’s and early 00’s will never be experienced again, but having said that, there are a number of forums that did in fact survive the social media onslaught, and still enjoy a large user base. Forums tend to be niche. One that immediately sticks out is Reddit - despite looking like it was designed in the 80s, it still has an enormous user base. Another is Stack Overflow, which needs no introduction. The key to any forum is the content it offers, and the more people whom contribute in terms of posting , the more popular and widely respected it becomes as a reliable source of information. Forums are still intensely popular with gamers, alongside those that offer tips on hacking etc.
  • Madchatthewundefined

    Bootstrap Version

    Solved Customisation nodebb bootstrap
    8
    5 Votes
    8 Posts
    1k Views
    Madchatthewundefined
    @phenomlab That will be nice once they have completed that. It will be interesting to see how long that takes. So for now I will use custom css to make it look the way I want. Frameworks just make things a little faster. Thanks @phenomlab
  • pwsincdundefined

    Fontawesome 5

    Unsolved Customisation fonts css nodebb
    14
    1 Votes
    14 Posts
    3k Views
    phenomlabundefined
    @pwsincd hi. Just following up on this thread (I know it’s old) but was curious to understand if it’s still an issue or not ?
  • crazycellsundefined

    background color of the footer area

    Solved Customisation nodebb
    7
    1
    7 Votes
    7 Posts
    1k Views
    crazycellsundefined
    @phenomlab thank you very much