You are viewing an offline version of MongoDB documentation. Some page features might be unavailable. To view the latest version of the page or use interactive features, visit the live page.

Database Manual

Time Series

Migrate Data into a Time Series Collection

To migrate data from an existing collection into a time series collection, use an $out stage in your aggregation pipeline.

Note

In MongoDB versions prior to 7.0.3, an aggregation pipeline cannot use $out to output to a time series collection. To migrate data into a time series collection with MongoDB versions prior to 7.0.3, use mongodump and mongorestore.

Migrate Data to a Time Series Collection

(Optional) Transform your data to create a metadata field if one doesn't exist. This field is not required.

If the original collection doesn't have a metadata field, use the $addFields aggregation stage to add it.

Consider a collection with weather data that uses the format:

 {
    "_id" : ObjectId("5553a998e4b02cf7151190b8"),
    "st" : "x+47600-047900",
    "ts" : ISODate("1984-03-05T13:00:00Z"),
    "position" : {
      "type" : "Point",
      "coordinates" : [ -47.9, 47.6 ]
    },
    "elevation" : 9999,
    "callLetters" : "VCSZ",
    "qualityControlProcess" : "V020",
    "dataSource" : "4",
    "type" : "FM-13",
    "airTemperature" : { "value" : -3.1, "quality" : "1" },
    "dewPoint" : { "value" : 999.9, "quality" : "9" },
    "pressure" : { "value" : 1015.3, "quality" : "1" },
    "wind" : {
      "direction" : { "angle" : 999, "quality" : "9" },
      "type" : "9",
      "speed" : { "rate" : 999.9, "quality" : "9" }
    },
    "visibility" : {
      "distance" : { "value" : 999999, "quality" : "9" },
      "variability" : { "value" : "N", "quality" : "9" }
    },
    "skyCondition" : {
      "ceilingHeight" : { "value" : 99999, "quality" : "9", "determination" : "9" },
      "cavok" : "N"
    },
    "sections" : [ "AG1" ],
    "precipitationEstimatedObservation" : { "discrepancy" : "2",
    "estimatedWaterDepth" : 999 }
}

The following pipeline stages add a metaData field and use $project to include or exclude the remaining fields in the document:

{ $addFields: {
    metaData: {
      "st": "$st",
      "position": "$position",
      "elevation": "$elevation",
      "callLetters": "$callLetters",
      "qualityControlProcess": "$qualityControlProcess",
      "type": "$type"
    }
  },
},
{ $project: {
    _id: 1,
    ts: 1,
    metaData: 1,
    dataSource: 1,
    airTemperature: 1,
    dewPoint: 1,
    pressure: 1,
    wind: 1,
    visibility: 1,
    skyCondition: 1,
    sections: 1,
    precipitationEstimatedObservation: 1
  }
}

Use the timeseries option with the $out aggregation stage

The example below uses the db.collection.aggregate() helper method. For the aggregation stage syntax, see $out. For a full explanation of the time series options, see the Time Series Field Reference.

db.weather_data.aggregate([
  {
     $addFields: {
       metaData: {
         "st": "$st",
         "position": "$position",
         "elevation": "$elevation",
         "callLetters": "$callLetters",
         "qualityControlProcess": "$qualityControlProcess",
         "type": "$type"
       }
     },
  }, {
     $project: {
        _id: 1,
        ts: 1,
        metaData: 1,
        dataSource: 1,
        airTemperature: 1,
        dewPoint: 1,
        pressure: 1,
        wind: 1,
        visibility: 1,
        skyCondition: 1,
        sections: 1,
        precipitationEstimatedObservation: 1
     }
  }, {
     $out: {
       db: "mydatabase",
       coll: "weathernew",
       timeseries: {
         timeField: "ts",
         metaField: "metaData"
       }
     }
  }
])

After you run this command, you have the weathernew collection below:

db.weathernew.findOne()
{
   "_id" : ObjectId("5553a998e4b02cf7151190b8"),
   "ts" : ISODate("1984-03-05T13:00:00Z"),
   "metaData" : {
     "st" : "x+47600-047900",
     "position" : {
       "type" : "Point",
       "coordinates" : [ -47.9, 47.6 ]
     },
     "elevation" : 9999,
     "callLetters" : "VCSZ",
     "qualityControlProcess" : "V020",
     "type" : "FM-13"
   },
   "dataSource" : "4",
   "airTemperature" : { "value" : -3.1, "quality" : "1" },
   "dewPoint" : { "value" : 999.9, "quality" : "9" },
   "pressure" : { "value" : 1015.3, "quality" : "1" },
   "wind" : {
     "direction" : { "angle" : 999, "quality" : "9" },
     "type" : "9",
     "speed" : { "rate" : 999.9, "quality" : "9" }
   },
   "visibility" : {
     "distance" : { "value" : 999999, "quality" : "9" },
     "variability" : { "value" : "N", "quality" : "9" }
   },
   "skyCondition" : {
     "ceilingHeight" : { "value" : 99999, "quality" : "9", "determination" : "9" },
     "cavok" : "N"
   },
   "sections" : [ "AG1" ],
   "precipitationEstimatedObservation" : { "discrepancy" : "2", "estimatedWaterDepth" : 999 }
}

If your original collection had secondary indexes, manually recreate them now. If your collection includes timeField values before 1970-01-01T00:00:00.000Z or after 2038-01-19T03:14:07.000Z, MongoDB logs a warning and disables some query optimizations that make use of the internal clustered index. Create a secondary index on the timeField to regain query performance and resolve the log warning.