How to add items to robots.txt with WordPress

If you wasn’t aware, WordPress auto-generates its own robots.txt file, and that there is a filter that you can hook into to allow you to add your own rules to the file.

The filter in question is robots_txt filter. Using this information I hand crafted this handy little snippet:

 * Filter function used to disallow pdfs from robots.txt
 * @param    string  $output     The output of the robots.txt file
 * @return   string  $public     If the site is public facing
function disallow_pdf_index( $output, $public ) {

    if ( '1' === $public ) {
        $output .= "Disallow: *.pdf\n";
    return $output;
add_filter( 'robots_txt', 'disallow_pdf_index', 0, 2 );

So there you go, a short, but handy little blog post detailing how you too can add entries to your robots.txt file programatically using WordPress.

Posted by Matt Watson

Matt Watson loves to talk about Health, Wealth and Code (mainly WordPress). Get in touch with Matt or follow Matt on Twitter to get notified about his latest posts.