Skip to content

Astro robots not generating a robots file or a meta tag #54

@gplusdotgr

Description

@gplusdotgr

Here's my astro config:

import { defineConfig } from "astro/config"
import node from "@astrojs/node"
import vue from "@astrojs/vue"
import svgLoader from "@gkatsanos/vite-svg-loader"
import tailwind from "@astrojs/tailwind"
import robotsTxt from "astro-robots-txt"

// https://astro.build/config
export default defineConfig({
  server: {
    port: 3000,
    host: "0.0.0.0",
  },
  output: "server",
  adapter: node({
    mode: "standalone",
  }),
  build: {
    assets: "requests-frontend/_astro",
  },
  vite: {
    server: {
      proxy: {
        "^/octopus/api/.*": {
          target: "https://www.wlw-staging.de",
          changeOrigin: true,
        },
      },
    },
    plugins: [
      svgLoader({
        defaultImport: "url", // or 'raw'
      }),
    ],

    ssr: {
      noExternal: ["path-to-regexp"],
    },
    optimizeDeps: {
      // Why? dd-trace attempts to patch graphql, but it's not a dependency of this project.
      exclude: ["graphql"],
    },
  },
  integrations: [
    vue(),
    tailwind(),
    robotsTxt({
      policy: [
        {
          userAgent: "*",
          // The next line enables or disables the crawling on the `robots.txt` level
          disallow: "/",
        },
      ],
    }),
  ],
})

Looking at the HTML and the network tab, dont see a robots file or a meta tag.. whats up?
btw my Astro app is served from a sub-directory ( www.example.com/myastroapp/ ) with an nginx webserver on top of it.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions