Automate Internal Linking at Scale using AI

Internal linking is a crucial aspect of search engine optimization (SEO) that helps search engines understand the structure and relevance of your website. By strategically linking related pages within your site, you can improve user navigation, distribute link equity, and boost your search engine rankings. However, manually creating and managing internal links can be a time-consuming and tedious process, especially for large websites with frequent content updates. This is where AI comes in – by automating internal linking, you can save time, ensure consistency, and scale your SEO efforts effortlessly.

‍

We've used Moonlit to automate the process of injecting internal links into a target blog post on any website which you can test in the app embed above. In this post, we'll discuss pain points, challenges, benefits to automation, and other aspects of internal linking. Lastly, we'll walkthrough how we built this app, and how you can run it at scale.

‍

Perks of Automating Internal Linking

Imagine having a tireless assistant that constantly analyzes your website's content, identifies relevant topics, and creates internal links accordingly. That's the power of AI-driven internal linking automation. Here are some key benefits:

1. Improved Efficiency and Time Savings: With AI handling the internal linking process, you can focus on creating high-quality content instead of spending hours manually adding links.
2. Consistent and Comprehensive Linking: AI algorithms can ensure that your internal linking is consistent across your website, without any missed opportunities or broken links.
3. Scalability: As your website grows and evolves, AI can easily adapt and scale your internal linking strategy to accommodate new content and changes.

Approaches to Automating Internal Linking with AI

One popular approach to automating internal linking is using topic modeling libraries like BERTopic. Topic modeling is a technique that allows you to identify and cluster similar documents based on their content. By applying topic modeling to your website's pages and blog posts, you can automatically group related content and create internal links between them.

However, it's important to note that topic modeling algorithms may sometimes produce uneven topic distributions or unexpected ordering of representative documents. This can require additional adjustments or testing to ensure the generated internal links are relevant and logical.

Data Preparation and Preprocessing

Before diving into topic modeling, you need to prepare your website's data. This involves collecting and organizing your content, such as pages, blog posts, and other relevant information. You may need to perform text cleaning and preprocessing techniques to remove noise and standardize the data.

One useful technique is extracting relevant information like titles and URLs using regular expressions or other string manipulation methods. This can help you efficiently process and analyze your content for internal linking purposes.

For our app, we used the pages meta title and description and performed semantic clustering using Moonlit's built-in clustering function.

Integrating with CMS or Website Framework

To fully automate internal linking, you'll need to integrate the AI-based process with your existing content management system (CMS) or website framework. This may involve leveraging CMS features, custom filters, or specific plugins to apply the generated internal links to your content. Moonlit also offer API access to Apps so if you have the technical expertise or have access to any integrations platform such as Zapier or Make, you can take your content pipeline to the next level.

Depending on your CMS or framework, you may need to handle order discrepancies or representation issues that arise during the integration process. It's crucial to test and refine the integration to ensure the automated internal linking works seamlessly with your website's architecture.

Validating and Refining the Automated Process

AI-driven internal linking automation is not a set-it-and-forget-it solution. It's essential to regularly review the generated internal links for accuracy and relevance. You may need to iterate and fine-tune the process based on feedback and performance metrics.

In some cases, human oversight or manual adjustments may be necessary to ensure the automated internal linking aligns with your SEO goals and user experience. Continuously monitor and optimize the process to achieve the best results.

Maintenance and Scalability

As your website grows and evolves, it's important to have strategies in place to keep your automated internal linking process up-to-date. This includes handling new content additions, website changes, and algorithm updates.

Scalability is another key consideration, especially for larger websites or content volumes. Our app currently handles up to 300 blog posts, but there are solutions to this limitation, you can reach out to us if you have such requirements.

Conclusion

Automating internal linking using AI is a game-changer for SEO and content professionals. By leveraging the power of topic modeling and content analysis, you can save time, improve consistency, and scale your internal linking efforts effortlessly.

However, it's important to approach AI-based automation with a critical eye and be prepared to handle potential challenges like uneven topic distributions or integration issues. Continuous optimization and adaptation are key to ensuring the success of your automated internal linking strategy.

Continue reading below to learn more about how we built our internal linking tool using Moonlit.

‍

How it works

‍

Imagine you have a website with hundreds of blog posts. Manually reading each post to find relevant sections for internal linking can be time-consuming. Here's how our tool simplifies the process: First, provide the URL of your website's sitemap. This allows us to access all your blog posts, as well as the URL of the specific post where you want to add internal links. Utilising Moonlit's advanced SEO capabilities, we begin by refining the sitemap URLs through semantic K-means Clustering. This process filters out irrelevant URLs, keeping only the pertinent ones. Next, we use a sophisticated Language Model, like GPT-4, which has proven effective in our experience. This model analyses your target blog post and the curated list of relevant posts. It then identifies sentences or sections in the target post where internal links from the list can be seamlessly integrated.

App Breakdown

All Apps in Moonlit consist of Inputs, Logic, and Outputs. Let's break down what we did for each section.

‍

Inputs

• Sitemap URL: This will be passed to the 'Extract Sitemap URLs' logic node to fetch the 'url', 'title', and 'description' of each page.
• Blogs Prefix: to filter out urls by specific path segments, if all blog posts are prefixed with /blog/{blog_title}, we can set the value to '/blog/' to filter for that.
• Blog Post URL: This is our target blog post where we will inject internal links into.

‍

Logic

• Step 1: In the first step of our logic we will be running two functions in parallel. The Extract Sitemap URLs node will return a table containing each post url, title, and description. We also run the Scrape Webpage with the Output Format option set to 'Main Text' to retrieve the post body text.
• Step 2: In this step we used a custom python snippet to make sure that our target blog post url is included in the table given by the Extract Sitemap URLs node. A quick caveat here, since we only have the text body and not the meta title and description, I've used the body in place of the description and the URL in place of the title which shouldn't negatively affect our next step.
• Step 3: The K-Means Clustering node is used with 'Text Clustering' ticked so that it converts all the text in our table to vector embeddings so that we can compute the distance between each one and place them into 5 separate buckets where each bucket contains the posts most relevant to each other.

‍

‍

• Step 4: As the last pre-processing step before prompting our large language model I filter out all the clusters keeping only the cluster that contains the url to our target blog post. I also drop the 'cluster' column since it's irrelevant to our prompt.
• Step 5: This is where the actual internal linking happens, we pass the List of Relevant Blog Posts along with the Target Blog Post Copy and ask GPT to add the internal links.

‍

‍

Here is the full prompt used for instructing the AI to inject internal links into the target post:

‍

Your task will be to write a comprehensive blog post discussing topics as specified in a content brief which I'll
provide. Along side the content brief, I will also provide you with a set of guidelines, images, and compiled 
research.You will need each of these 3 section to successfully write a captivating post. The title for this blog 
post is "{{Title}}".

-----
# Content Brief
{{Outline}}
-----
# Compiled Research
{{Compiled Research}}
-----
# Images
{{Images Desc}}
-----
# Guidelines
- The blog post must be in markdown format, don't start with ```text, just use markdown from the get go.
- Avoid overused and cliche opening statements like "In today's past paced digital landscaped", be more creative 
with these opening statements.
- Avoid using unneccessary technical jargon.
- The reader persona is a content or seo specialist with an interest in AI for streamlining content operations.
- Include source links throughout the text where suitable, from the research section.
- Add Images throughout the blog post whenever relevant, the images in the 'Images' section are programmatically 
fetched and it is plausable that some are irrelevant so only use them if they are relevant to the section you're 
writing, if none are relevant just don't include any images.
- The search results may sometimes include information about a product that assists with the given topic, I don't 
want you to promote any other SaaS, you can use the information they provide about it but don't promote it.

That's it! Please continue with writing in post in markdown format.

‍

Outputs

• Original Post: The original post body without links just for direct comparison.
• Improved Post: The final result, containing the improved post with internal links.

‍

Testing & Results

Now that everything in the 'build' section is complete we can test the app. I've used Notion as an example. It took some minor refinements to the prompt and testing with different models until I've landed good results. The model was able to return to the same text body with 3 relevant internal links injected throughout. The default values we used are loaded with the app so you can test yourself.

‍

Injecting Internal Links at Scale

Running this app at scale can be done using Moonlit's bulk run feature. You'll first need to prepare a CSV table with 3 columns, each column will correspond to an input, so for example:

‍

The only column we're changing is the target blog post column since the sitemap or blog prefix should just stay the same as long as we are working for the same website. Of course, you can also run this for multiple websites at scale, just think of each row in your table as a seperate app run.

‍

Once you have the CSV file ready, and assuming you have a Moonlit account if not create one here, you can make a clone of our app to your project, this can be done through the button on the top right corner from the app share link here or directly from our templates library.

‍

After you've cloned the app, you can now go to the Bulk Runs tab from the panel at the top; from there just create a new job, upload the csv you created earlier, map the inputs to columns, select the 'Result' to be part of the included outputs and once the data is loaded you can start the job. You'll be notified by email once the job is complete. Read more about Bulk Runs here