3637814 1835090 1835090 2064386 3670564 3707731 3707732 3707733 3604634 3604636 3604638 3604639 3604640 4849685 5 2008-05-02 08:15:36.643 2008-05-02 08:43:44.513 3637815 3637817 3637819 3637821 17858815 1835090 1835090 19038486 2064386 17891559 18777311 5 2011-07-07 20:40:04.910 2012-02-03 16:41:13.737 17858817 17858819 18579806 18579809 17825865 2162859 2064386 2195625 2293799 2293800 2293801 9 2007-11-06 16:28:36.237 2007-11-07 09:28:44.627 2162860 2162861 2162862 2162863 2162864 2162865 2162866 2162867 2064386 1835089 1835090 4816897 4816898 4816899 4816900 4816901 4816902 4816903 19300604 34406571 34406572 34406573 34406574 34406575 34406576 34406577 34406578 34406579 34406580 34406581 34406582 34406583 34406584 2007-10-03 12:37:04.967 2012-02-03 09:27:07.373 global 2097157 2]]> 2097156 theme.key ]]> 1835140 1835090 1835090 2064386 1867904 18777176 18777177 18777178 1933374 1933375 1933376 1933377 1933379 1933382 2293795 2293797 3604633 13 2007-10-05 14:24:44.267 2012-02-03 09:25:50.720 1835141 1835142 1835143 1835145 1835147 1835149 1835150 1835151 1835153 2162855 2162857 18579784 16777265 1835090 1835090 2064386 16810033 13 2011-04-15 06:27:16.287 2014-07-21 20:44:53.467 16777266 16777268 16777269 16777277 16777279 16777283 18252364 26640533 26640535 26640537 26640539 26640540 16973840 16973841 16973842 16973843 16973844 16973845 16973846 16973847 16973848 16973849 16973850 16973851 16973852 16973853 16973854 16973855 16973856 16973825 16973826 16973827 16973828 16973829 16973830 26837003 26837004 26837005 26837006 16973831 16973832 16973833 16973834 16973835 16973836 16973837 16973838 16973839 16973857 2097158 O3S false global ]]> 1835120 1835090 1835090 2064386 1867884 1933362 1933364 1933365 2293763 2293764 1 2007-10-03 16:12:37.030 2007-10-03 16:12:37.030 18579793 1835090 18579795 1835090 2064386 18645308 18777237 1 2012-02-03 12:01:15.480 2012-02-03 12:01:15.480 18579795 18579793 1835090 18579793 2064386 18645310 18777238 18777239 18777240 18777241 18777242 2 2012-02-03 12:04:08.647 2012-02-03 12:17:48.607 18579797 18841670 18841671 18841672 18841673 18841674 17858807 1835090 1835090 2064386 17891551 2 2011-07-07 12:34:06.707 2011-07-07 15:21:24.070 17858810 9798591 1835090 1835090 2064386 9831335 1 2009-05-18 09:01:01.490 2009-05-18 09:01:01.490 2162834 1835090 1835090 3637249 3637250 3637251 2064386 2195600 33292305 33292306 33292307 33292308 33292309 33292310 33292311 33292312 33292313 33292314 33292315 33292316 3604481 3604483 3604513 28180486 74 2007-10-29 10:49:47.743 2021-06-14 16:10:18.180 2162837 2162838 2162839 2162840 2162842 2162843 2162844 2162852 2162854 2162856 2162871 2162888 2162925 2162926 2162928 2162929 3114498 6259082 6259084 6259086 6259091 6259111 6259114 6259252 6259254 6259256 6259316 6259318 6259320 7766129 9372106 9372126 9372155 9372156 9797946 9797971 9798092 9798154 9798168 10356341 10911784 10911785 10911787 11240264 11240266 11240272 11240442 11240443 11240445 11796614 11796616 11796617 11796677 11796683 11796685 11796687 15631231 15631233 15631235 15631237 15631239 15631240 15631247 17236075 17236076 17236077 18579771 18579959 18579961 18579962 18580006 33095686 33095687 1835112 1835090 1835090 2064386 1867876 1901044 1933355 1933357 1 2007-10-03 14:02:09.923 2007-10-03 14:02:09.923 3114486 1835090 1835090 2064386 3179957 3 2008-04-02 11:57:23.537 2012-02-03 16:38:46.177 3114488 18579801 3309673 1835090 1835112 1835120 1835140 2162834 3114486 3637814 9798591 16777265 17858807 17858815 18579793 2064386 1867854 18777298 18777299 18777300 18777301 18777302 18777303 18777304 18777305 18777306 18777307 18777308 18777309 18777310 1933327 1933328 1933332 1933333 1933335 1933336 1933338 1933341 1933343 1933344 1933345 1933346 1933348 1933349 1933350 1933351 1933353 1933354 1933358 1933363 1933378 1933380 1933381 2293761 2293794 2293796 2293798 2293897 3604637 62 2007-10-03 12:37:04.993 2012-02-03 16:41:13.727 1835096 1835098 1835099 1835100 1835101 1835102 1835103 1835104 1835113 1835114 1835119 1835121 1835122 1835123 1835124 1835125 1835126 1835127 1835128 1835129 1835130 1835131 1835132 1835133 1835134 1835135 1835136 1835137 1835138 1835139 1835144 1835146 1835148 1835152 2162698 2162853 3637823 9797895 9797896 16777281 17858805 17858813 18579768 18579770 18579775 18579777 18579778 18579779 18579780 18579782 18579783 18579785 18579786 18579788 18579790 18579791 18579802 18579804 18579805 18579807 18579808 9928721 1835135 1867899 26 2007-10-03 12:37:04.993 2007-10-03 16:34:19.473 1835090 2162863 2195629 4 2007-11-06 16:28:36.237 2007-11-06 16:30:17.300 2162859 1835136 1867900 27 2007-10-03 12:37:04.993 2007-10-03 16:41:02.103 1835090 9798168 9830922 39 2007-10-29 10:49:47.743 2009-05-04 12:54:39.500 2162834 2162864 2195630 5 2007-11-06 16:28:36.237 2007-11-06 16:31:28.897 2162859 3604483 1 2162834 2008-04-11 08:28:15.053 2008-04-11 08:28:15.053 1835137 1867901 28 2007-10-03 12:37:04.993 2007-10-04 10:28:13.733 1835090 2162865 2195631 6 2007-11-06 16:28:36.237 2007-11-06 16:31:55.103 2162859 1835138 1867902 29 2007-10-03 12:37:04.993 2007-10-04 10:32:15.457 1835090 2162866 2195632 7 2007-11-06 16:28:36.237 2007-11-06 16:38:35.127 2162859 1835131 1867895 22 2007-10-03 12:37:04.993 2007-10-03 16:32:25.597 1835090 1835132 1867896 23 2007-10-03 12:37:04.993 2007-10-03 16:32:42.727 1835090 2162860 2195626 1 2007-11-06 16:28:36.237 2007-11-06 16:28:36.237 2162859 1835133 1867897 24 2007-10-03 12:37:04.993 2007-10-03 16:32:54.100 1835090 2162861 2195627 2 2007-11-06 16:28:36.237 2007-11-06 16:29:08.927 2162859 1835134 1867898 25 2007-10-03 12:37:04.993 2007-10-03 16:33:24.667 1835090 2162862 2195628 3 2007-11-06 16:28:36.237 2007-11-06 16:29:45.570 2162859 1835143 1867907 3 2007-10-05 14:24:44.267 2007-10-05 14:35:45.217 1835140 2162871 2195637 11 2007-10-29 10:49:47.743 2007-11-06 16:19:29.250 2162834 1835144 1867908 31 2007-10-03 12:37:04.993 2007-10-05 14:19:57.550 1835090 1835145 1867909 4 2007-10-05 14:24:44.267 2007-10-05 14:36:11.650 1835140 1835146 1867910 32 2007-10-03 12:37:04.993 2007-10-05 14:38:57.210 1835090 1835139 1867903 30 2007-10-03 12:37:04.993 2007-10-04 10:53:55.393 1835090 2162867 2195633 8 2007-11-06 16:28:36.237 2007-11-07 09:27:54.463 2162859 1835141 1867905 1 2007-10-05 14:24:44.267 2007-10-05 14:24:44.267 1835140 1835142 1867906 2 2007-10-05 14:24:44.267 2007-10-05 14:31:31.527 1835140 3604481 1 2162834 2008-04-11 08:27:15.047 2008-04-11 08:27:15.047 33292307 2162834 2021-06-14 16:10:18.227 2021-06-14 16:10:18.227 1835119 1867883 11 2007-10-03 12:37:04.993 2007-10-03 14:04:37.560 1835090 33292306 2162834 2021-06-14 16:10:18.227 2021-06-14 16:10:18.227 1835122 1867886 13 2007-10-03 12:37:04.993 2007-10-03 16:14:49.213 1835090 33292305 2162834 2021-06-14 16:10:18.227 2021-06-14 16:10:18.227 1835121 1867885 12 2007-10-03 12:37:04.993 2007-10-03 16:12:18.260 1835090 1835128 1867892 19 2007-10-03 12:37:04.993 2007-10-03 16:31:23.457 1835090 18777241 18579795 2012-02-03 12:17:48.630 2012-02-03 12:17:48.630 2162888 2195654 12 2007-10-29 10:49:47.743 2007-11-07 11:09:32.407 2162834 1835127 1867891 18 2007-10-03 12:37:04.993 2007-10-03 16:31:10.700 1835090 33095687 33161223 73 2007-10-29 10:49:47.743 2021-06-14 16:09:39.617 2162834 6259082 6226300 18 2007-10-29 10:49:47.743 2008-04-03 20:17:25.087 2162834 18777240 18579795 2012-02-03 12:17:48.630 2012-02-03 12:17:48.630 1835130 1867894 21 2007-10-03 12:37:04.993 2007-10-03 16:31:59.423 1835090 1835129 1867893 20 2007-10-03 12:37:04.993 2007-10-03 16:31:40.740 1835090 18777242 18579795 2012-02-03 12:17:48.630 2012-02-03 12:17:48.630 1835124 1867888 15 2007-10-03 12:37:04.993 2007-10-03 16:28:06.237 1835090 18777237 18579793 2012-02-03 12:01:15.530 2012-02-03 12:01:15.530 1835123 1867887 14 2007-10-03 12:37:04.993 2007-10-03 16:18:25.353 1835090 18777239 18579795 2012-02-03 12:17:48.630 2012-02-03 12:17:48.630 33095686 33161222 72 2007-10-29 10:49:47.743 2012-03-01 08:11:15.897 2162834 1835126 1867890 17 2007-10-03 12:37:04.993 2007-10-03 16:29:18.180 1835090 18777238 18579795 2012-02-03 12:17:48.630 2012-02-03 12:17:48.630 1835125 1867889 16 2007-10-03 12:37:04.993 2007-10-03 16:28:54.587 1835090 9928721 1835090 2009-04-29 12:35:51.870 2009-04-29 12:35:51.870 39936 1 33292316 2162834 2021-06-14 16:10:18.227 2021-06-14 16:10:18.227 6259084 6226302 19 2007-10-29 10:49:47.743 2008-09-05 09:19:29.207 2162834 6259086 6226304 20 2007-10-29 10:49:47.743 2008-09-05 10:35:07.903 2162834 33292312 2162834 2021-06-14 16:10:18.227 2021-06-14 16:10:18.227 33292313 2162834 2021-06-14 16:10:18.227 2021-06-14 16:10:18.227 16810033 16777265 33292314 2162834 2021-06-14 16:10:18.227 2021-06-14 16:10:18.227 33292315 2162834 2021-06-14 16:10:18.227 2021-06-14 16:10:18.227 33292308 2162834 2021-06-14 16:10:18.227 2021-06-14 16:10:18.227 33292309 2162834 2021-06-14 16:10:18.227 2021-06-14 16:10:18.227 6259091 6226309 21 2007-10-29 10:49:47.743 2008-09-05 11:44:19.407 2162834 33292310 2162834 2021-06-14 16:10:18.227 2021-06-14 16:10:18.227 33292311 2162834 2021-06-14 16:10:18.227 2021-06-14 16:10:18.227 9798154 9830908 38 2007-10-29 10:49:47.743 2009-05-01 09:32:48.407 2162834 1835153 1867917 9 2007-10-05 14:24:44.267 2007-10-05 15:07:20.360 1835140 1835152 1867916 34 2007-10-03 12:37:04.993 2007-10-05 14:42:05.107 1835090 9831335 >/oasis/iridium/iri dium1/$DT rm /oasis/iridium/iridium1/*.sbd fi cat /oasis/iridium/iridium1/textfile* >>/oasis/iridium/iridium1/text$DT rm /oasis/iridium/iridium1/textfile* rm /oasis/iridium/iridium1/incoming /bin/cp /dev/null /var/mail/iridium1 fi {noformat}]]> 9798591 1835151 1867915 8 2007-10-05 14:24:44.267 2007-10-05 14:51:11.567 1835140 1835150 1867914 7 2007-10-05 14:24:44.267 2007-10-05 14:49:45.767 1835140 1835149 1867913 6 2007-10-05 14:24:44.267 2007-10-05 14:40:58.863 1835140 1835148 1867912 33 2007-10-03 12:37:04.993 2007-10-05 14:40:22.797 1835090 1835147 1867911 5 2007-10-05 14:24:44.267 2007-10-05 14:39:39.500 1835140 6259111 6226328 22 2007-10-29 10:49:47.743 2008-09-05 16:47:55.890 2162834 6259114 6226331 23 2007-10-29 10:49:47.743 2008-09-08 11:46:13.127 2162834 18777309 1835090 2012-02-03 16:41:13.737 2012-02-03 16:41:13.737 18777308 1835090 2012-02-03 16:41:13.737 2012-02-03 16:41:13.737 18777311 17858815 2012-02-03 16:41:13.743 2012-02-03 16:41:13.743 18777310 1835090 2012-02-03 16:41:13.737 2012-02-03 16:41:13.737 15631247 15663997 63 2007-10-29 10:49:47.743 2011-01-31 16:35:23.767 2162834 1901044 1835112 2007-10-03 14:02:10.007 2007-10-03 14:02:10.007 18579959 18645467 68 2007-10-29 10:49:47.743 2012-02-03 09:16:05.513 2162834 18579962 18645470 70 2007-10-29 10:49:47.743 2012-02-28 12:23:09.070 2162834 18579961 18645469 69 2007-10-29 10:49:47.743 2012-02-28 12:12:17.610 2162834 18777301 1835090 2012-02-03 16:41:13.737 2012-02-03 16:41:13.737 18777300 1835090 2012-02-03 16:41:13.737 2012-02-03 16:41:13.737 18777303 1835090 2012-02-03 16:41:13.737 2012-02-03 16:41:13.737 18777302 1835090 2012-02-03 16:41:13.737 2012-02-03 16:41:13.737 18777305 1835090 2012-02-03 16:41:13.737 2012-02-03 16:41:13.737 18777304 1835090 2012-02-03 16:41:13.737 2012-02-03 16:41:13.737 18777307 1835090 2012-02-03 16:41:13.737 2012-02-03 16:41:13.737 18777306 1835090 2012-02-03 16:41:13.737 2012-02-03 16:41:13.737 18580006 18645514 71 2007-10-29 10:49:47.743 2012-02-28 12:23:38.710 2162834 3309673 3114486 2008-04-02 11:58:05.017 2008-04-02 11:58:05.017 118329 1 3114498 3179968 17 2007-10-29 10:49:47.743 2007-11-16 14:05:58.687 2162834 18777298 1835090 2012-02-03 16:41:13.737 2012-02-03 16:41:13.737 18777299 1835090 2012-02-03 16:41:13.737 2012-02-03 16:41:13.737 15631233 15663984 58 2007-10-29 10:49:47.743 2011-01-31 16:24:04.077 2162834 15631231 15663982 57 2007-10-29 10:49:47.743 2010-04-06 11:10:57.857 2162834 2162837 2195603 1 2007-10-29 10:49:47.743 2007-10-29 10:49:47.743 2162834 15631237 15663988 60 2007-10-29 10:49:47.743 2011-01-31 16:27:21.090 2162834 2162838 2195604 2 2007-10-29 10:49:47.743 2007-10-29 10:52:13.493 2162834 7766129 7798893 30 2007-10-29 10:49:47.743 2008-10-08 14:59:56.053 2162834 15631235 15663986 59 2007-10-29 10:49:47.743 2011-01-31 16:25:52.380 2162834 2162842 2195608 5 2007-10-29 10:49:47.743 2007-10-29 13:44:18.087 2162834 15631240 15663991 62 2007-10-29 10:49:47.743 2011-01-31 16:34:39.300 2162834 2162839 2195605 3 2007-10-29 10:49:47.743 2007-10-29 11:04:37.773 2162834 15631239 15663990 61 2007-10-29 10:49:47.743 2011-01-31 16:33:24.957 2162834 28180486 1 2162834 2015-08-27 10:26:15.050 2015-08-27 10:26:15.050 2162840 2195606 4 2007-10-29 10:49:47.743 2007-10-29 12:39:03.793 2162834 2162844 2195610 7 2007-10-29 10:49:47.743 2007-10-30 13:57:14.710 2162834 2162843 2195609 6 2007-10-29 10:49:47.743 2007-10-29 15:43:04.510 2162834 2162854 2195620 9 2007-10-29 10:49:47.743 2007-11-06 15:42:37.510 2162834 2162853 2195619 36 2007-10-03 12:37:04.993 2007-10-16 12:15:03.307 1835090 3114488 3179959 1 2008-04-02 11:57:23.537 2008-04-02 11:57:23.537 3114486 2162852 2195618 8 2007-10-29 10:49:47.743 2007-10-30 16:15:00.757 2162834 2162857 2195623 11 2007-10-05 14:24:44.267 2007-11-06 16:16:15.747 1835140 2162856 2195622 10 2007-10-29 10:49:47.743 2007-11-06 16:13:38.160 2162834 2162855 2195621 10 2007-10-05 14:24:44.267 2007-10-05 15:10:00.637 1835140 34406571 2064386 2026-02-04 08:34:45.257 2026-02-04 08:34:45.257 34406572 2064386 2026-02-04 08:34:59.107 2026-02-04 08:34:59.107 1867884 1835120 34406579 2064386 2026-02-04 08:34:59.108 2026-02-04 08:34:59.108 34406580 2064386 2026-02-04 08:34:59.108 2026-02-04 08:34:59.108 34406577 2064386 2026-02-04 08:34:59.108 2026-02-04 08:34:59.108 34406578 2064386 2026-02-04 08:34:59.108 2026-02-04 08:34:59.108 34406575 2064386 2026-02-04 08:34:59.108 2026-02-04 08:34:59.108 1867876 1835112 34406576 2064386 2026-02-04 08:34:59.108 2026-02-04 08:34:59.108 34406573 2064386 2026-02-04 08:34:59.107 2026-02-04 08:34:59.107 34406574 2064386 2026-02-04 08:34:59.108 2026-02-04 08:34:59.108 1867904 1835140 11240264 11273004 44 2007-10-29 10:49:47.743 2009-10-20 10:06:09.023 2162834 11240266 11273006 45 2007-10-29 10:49:47.743 2010-03-05 10:00:16.680 2162834 9372155 9404896 33 2007-10-29 10:49:47.743 2009-03-09 11:49:00.577 2162834 9372156 9404897 34 2007-10-29 10:49:47.743 2009-03-09 15:43:48.720 2162834 11240272 11273012 46 2007-10-29 10:49:47.743 2010-03-05 10:04:28.970 2162834 1867854 1835090 1933354 1 1835090 2007-10-03 14:02:15.023 2007-10-03 14:02:15.023 18252364 18285122 7 2011-04-15 06:27:16.287 2011-04-15 09:47:07.843 16777265 17858817 17891561 1 2011-07-07 20:40:04.910 2011-07-07 20:40:04.910 17858815 1933355 1 1835112 2007-10-03 14:02:15.027 2007-10-03 14:02:15.027 1933357 3 1835112 2007-10-03 14:03:15.023 2007-12-19 12:53:15.027 17858813 17891557 42 2007-10-03 12:37:04.993 2011-07-07 12:32:05.323 1835090 1933358 1 1835090 2007-10-03 14:04:15.023 2007-10-03 14:04:15.023 17825865 17858815 2011-07-07 20:40:42.443 2011-07-07 20:40:42.443 23552 1 1933362 1 1835120 2007-10-03 16:13:15.017 2007-10-03 16:13:15.017 1933363 4 1835090 2007-10-03 16:13:15.020 2008-05-02 08:45:15.020 1933364 1 1835120 2007-10-03 16:13:15.023 2007-10-03 16:13:15.023 1933365 6 1835120 2007-10-03 16:13:15.027 2008-05-02 08:45:15.027 17858819 17891563 2 2011-07-07 20:40:04.910 2011-07-07 20:41:45.867 17858815 17891551 17858807 1933338 3 1835090 2007-10-03 13:19:15.020 2007-10-03 13:57:15.027 1933341 2 1835090 2007-10-03 13:26:15.027 2007-10-03 16:15:15.030 1933343 1 1835090 2007-10-03 13:26:15.033 2007-10-03 13:26:15.033 18777176 1835140 2012-02-03 09:25:50.740 2012-02-03 09:25:50.740 18777177 1835140 2012-02-03 09:25:50.740 2012-02-03 09:25:50.740 1933345 1 1835090 2007-10-03 13:32:15.020 2007-10-03 13:32:15.020 18777178 1835140 2012-02-03 09:25:50.740 2012-02-03 09:25:50.740 2162698 2195465 35 2007-10-03 12:37:04.993 2007-10-05 15:10:00.633 1835090 1933344 1 1835090 2007-10-03 13:27:15.017 2007-10-03 13:27:15.017 17858810 17891554 1 2011-07-07 12:34:06.707 2011-07-07 12:34:06.707 17858807 1933346 20 1835090 2007-10-03 13:32:15.023 2008-05-02 08:48:15.040 1933349 1 1835090 2007-10-03 13:35:15.017 2007-10-03 13:35:15.017 1933348 2 1835090 2007-10-03 13:35:15.013 2007-10-03 13:40:15.017 1933351 3 1835090 2007-10-03 13:40:15.027 2007-10-10 09:10:15.053 17858805 17891549 41 2007-10-03 12:37:04.993 2011-04-15 09:45:44.547 1835090 1933350 1 1835090 2007-10-03 13:40:15.023 2007-10-03 13:40:15.023 1933353 1 1835090 2007-10-03 14:01:15.020 2007-10-03 14:01:15.020 1835114 1867878 10 2007-10-03 12:37:04.993 2007-10-03 14:03:53.797 1835090 2293761 1 1835090 2007-10-10 09:10:15.043 2007-10-10 09:10:15.043 1835113 1867877 9 2007-10-03 12:37:04.993 2007-10-03 13:29:44.357 1835090 1835102 1867866 6 2007-10-03 12:37:04.993 2007-10-03 13:25:19.313 1835090 1835101 1867865 5 2007-10-03 12:37:04.993 2007-10-03 13:19:35.230 1835090 2293764 1 1835120 2007-10-10 13:10:15.030 2007-10-10 13:10:15.030 1835100 1867864 4 2007-10-03 12:37:04.993 2007-10-03 13:18:50.080 1835090 2293763 3 1835120 2007-10-10 13:09:15.023 2007-10-10 13:11:15.017 1835099 1867863 3 2007-10-03 12:37:04.993 2007-10-03 13:15:36.680 1835090 17236077 17268842 66 2007-10-29 10:49:47.743 2011-06-01 16:36:13.000 2162834 1835104 1867868 8 2007-10-03 12:37:04.993 2007-10-03 13:29:04.477 1835090 17236076 17268841 65 2007-10-29 10:49:47.743 2011-06-01 16:34:40.200 2162834 1835103 1867867 7 2007-10-03 12:37:04.993 2007-10-03 13:28:07.080 1835090 17236075 17268840 64 2007-10-29 10:49:47.743 2011-02-01 09:10:15.647 2162834 18645310 18579795 18645308 18579793 1933377 2 1835140 2007-10-05 14:40:15.020 2007-10-05 14:40:15.020 17891559 17858815 1933376 1 1835140 2007-10-05 14:40:15.017 2007-10-05 14:40:15.017 1835098 1867862 2 2007-10-03 12:37:04.993 2007-10-03 13:13:58.883 1835090 1835089 2064386 1867853 3 2007-10-03 12:37:04.963 2012-02-03 09:27:07.373 1835105 18579787 29523973 30867463 1933375 12 1835140 2007-10-05 14:32:15.017 2008-05-02 08:14:15.047 1933374 1 1835140 2007-10-05 14:25:15.023 2007-10-05 14:25:15.023 1835096 1867860 1 2007-10-03 12:37:04.993 2007-10-03 12:37:04.993 1835090 1933381 9 1835090 2007-10-05 15:10:15.030 2007-11-06 16:15:15.027 1933380 2 1835090 2007-10-05 15:10:15.027 2007-10-05 15:10:15.027 1933379 40 1835140 2007-10-05 14:41:15.030 2008-05-02 08:12:15.023 1933378 3 1835090 2007-10-05 14:41:15.023 2007-10-05 15:08:15.027 1933382 1 1835140 2007-10-05 15:12:15.013 2007-10-05 15:12:15.013 4816900 2064386 2008-06-03 11:01:08.433 2008-06-03 11:01:08.433 11796677 11829436 53 2007-10-29 10:49:47.743 2010-04-02 10:08:04.107 2162834 4816899 2064386 2008-06-03 11:01:08.433 2008-06-03 11:01:08.433 4816898 2064386 2008-06-03 11:01:08.433 2008-06-03 11:01:08.433 4816897 2064386 2008-06-03 11:00:58.290 2008-06-03 11:00:58.290 3604634 1 3637814 2008-05-02 08:16:15.023 2008-05-02 08:16:15.023 3604636 4 3637814 2008-05-02 08:21:15.030 2008-05-02 08:44:15.030 3604637 3 1835090 2008-05-02 08:26:15.037 2008-05-02 08:47:15.020 3604638 3 3637814 2008-05-02 08:35:15.020 2008-05-02 08:49:15.017 3604639 2 3637814 2008-05-02 08:38:15.030 2008-05-02 08:38:15.030 3604640 1 3637814 2008-05-02 11:34:15.057 2008-05-02 11:34:15.057 2293897 2 1835090 2007-12-19 12:55:15.027 2007-12-19 12:55:15.027 4816903 2064386 2008-06-03 11:01:08.437 2008-06-03 11:01:08.437 4816901 2064386 2008-06-03 11:01:08.433 2008-06-03 11:01:08.433 4816902 2064386 2008-06-03 11:01:08.437 2008-06-03 11:01:08.437 3670564 3637814 6259256 6226466 26 2007-10-29 10:49:47.743 2008-09-30 12:52:01.997 2162834 26640539 26673296 11 2011-04-15 06:27:16.287 2014-07-21 15:06:32.530 16777265 26640540 26673297 12 2011-04-15 06:27:16.287 2014-07-21 15:08:08.857 16777265 6259252 6226462 24 2007-10-29 10:49:47.743 2008-09-08 13:32:01.407 2162834 3637250 2162834 3670018 1 2008-04-11 09:40:14.197 2008-04-11 09:40:14.197 9798092 9830848 37 2007-10-29 10:49:47.743 2009-04-30 09:59:36.093 2162834 3637249 2162834 3670017 1 2008-04-11 09:37:55.507 2008-04-11 09:37:55.507 6259254 6226464 25 2007-10-29 10:49:47.743 2008-09-30 12:30:06.547 2162834 3637251 2162834 3670019 3702785 1 2008-04-11 10:09:26.827 2008-04-11 10:09:26.827 26640533 26673290 8 2011-04-15 06:27:16.287 2011-09-27 17:07:10.363 16777265 26640537 26673294 10 2011-04-15 06:27:16.287 2014-07-21 15:05:37.410 16777265 26640535 26673292 9 2011-04-15 06:27:16.287 2014-07-21 14:50:19.417 16777265 3179957 3114486 18841674 18579795 2012-02-03 12:05:55.423 2012-02-03 12:05:55.423 179795 1 11796683 11829442 54 2007-10-29 10:49:47.743 2010-04-06 10:37:50.593 2162834 11796685 11829444 55 2007-10-29 10:49:47.743 2010-04-06 10:57:48.203 2162834 18841671 18579795 2012-02-03 12:05:22.957 2012-02-03 12:05:22.957 178009 1 18841670 18579795 2012-02-03 12:05:12.660 2012-02-03 12:05:12.660 177301 1 11796687 11829446 56 2007-10-29 10:49:47.743 2010-04-06 11:06:05.993 2162834 18841673 18579795 2012-02-03 12:05:46.727 2012-02-03 12:05:46.727 162966 1 18841672 18579795 2012-02-03 12:05:32.747 2012-02-03 12:05:32.747 185506 1 3707733 3637814 2008-05-02 08:43:44.520 2008-05-02 08:43:44.520 11796617 11829376 52 2007-10-29 10:49:47.743 2010-04-02 10:04:21.063 2162834 3707732 3637814 2008-05-02 08:43:44.520 2008-05-02 08:43:44.520 11796616 11829375 51 2007-10-29 10:49:47.743 2010-04-02 10:00:23.907 2162834 11796614 11829373 50 2007-10-29 10:49:47.743 2010-03-17 11:47:56.387 2162834 19038486 17858815 19071252 19138271 2 2012-04-16 13:52:05.600 2012-04-16 13:52:36.423 19038487 2195600 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs and the path to the xml files to the newly deployed deviceIDs (aka ISI_IDs) and new xml directory. Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen or JEdit. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml file with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. \[Note: With the M1 - October 2009 deployment we added the individual inductive modem microcats as children of the mooring and also configured a parallel deployment of the TString that produced the same data. With the M2 - April 2010 deployment we configured the individual IM mirocrocats and did not configure a parallel TString deployment. This is a better model for the system and is much easier to configure and produces more easily consumed data by downstream processes such as combineTS.pl.\] With future deployments we will not configure TString. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|SSDS:Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[SSDS:main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /mbari/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. \\ For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed (or...as below \-rschramm 4/2010) \\ \\ {noformat} declare @myID as bigint set @myID = 31548 SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE id = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk in ( SELECT id FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL) ORDER BY id {noformat} h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSDS ingest will recognize them as new. # +Do not close (set endDates) the existing Test deployment until after the oasisToSSDS has run with the new platform deployment name+ and all new instrument deployments have been created in SSDS_Metadata. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadata, e.g.: {noformat} SELECT id, name, startDate, endDate, ParentID_FK FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') ORDER BY id DESC {noformat} Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file. # You will also need to edit the DStoNetCDF.pl script to set the new name of the M1 deployment, e.g.: {noformat} # # SSDS mooring name Deployment lookup # my %ssdsMooringDeplNames = ( M0 => 'M0', M1 => 'M1 - October 2009', M1Test => 'Test M1 - October 2009', M2 => 'M2 - April 2009', {noformat} # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. # When the mooring log message is sent to oasis with the actual time of deployment enter that as the startDate in the database.  All times (DTGs) in the SSDS_Metadata database are GMT. Make sure to set the startDate for the imctd microcat sensors too. This can be done with a query like this where you use the proper dataProduceIDs for the mooring and the imcd: {noformat} SELECT id, name, ParentID_FK, startDate, endDate, nominalDepth FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') AND (ParentID_FK = 31564) OR (ParentID_FK = 31548) ORDER BY id DESC {noformat} # Because of some bug in SSDS ingest the dataContainerTypes of the outputs from the instrument deployments do not get properly assigned the values of 'Stream'.  This needs to be fixed so that the NDBC datatransfers will work. To fix it edit the SSDS_Metadata database starting with a query like this: {noformat} SELECT ssdsdba.DataContainer.* FROM ssdsdba.DataProducer INNER JOIN ssdsdba.DataContainer ON ssdsdba.DataProducer.id = ssdsdba.DataContainer.DataProducerID_FK WHERE (ssdsdba.DataProducer.ParentID_FK = 31548) {noformat} where you use the DataProducerID for the new mooring deployment in the WHERE clause. ==> Change all of the 'File's in the dataContainerType' field to 'Stream's. h2. C. Procedures to be done after the mooring turn h4. Set up download info deployment and reporting Though not a real instrument, we configure a virtual 'dlinfo' instrument for the download scripts to attach download statistics data. We re-use the same device IDs for the M1 and M2 moorings, so it's best to configure this after the new mooring is out and all of those deployments have been closed. As the data are delivered "out of band" from the typical OASIS instruments we need to create a deployment for dlinfo instrument by hand. # The most direct way is to use the createDuplicateDeepDeployment service call. For example, to duplicate the 2009 M2 dlinfo deployment for the 2010 M2 deployment this call was executed: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer\|id=*34496*&p2Type=Date&p2Value=*2010-04-03T22:00:00Z*&p3Type=boolean&p3Value=false&p4Type=Date&p4Value=*2010-04-03T23:00:00Z*&p5Type=String&p5Value=getM2-download&p6Type=String&p6Value=&delimiter=\|. Of course you will need to adjust the DataProducer ID Date values for the new deployment you are creating. Here is the Key to the parameters: {noformat} Key: ---- http://localhost:8080/servlet/MetadataAccessServlet ?responseType=text &delimiter=| &objectToInvokeOn=DataProducerAccess &method=createDuplicateDeepDeployment &p1Type=DataProducer &p1Value=DataProducer|id=XXXX (XXXX is the ID of the deployment to copy) &p2Type=Date &p2Value=XXXXXX (XXXXXX is the start date of the new copy in XML format YYYY-MM-DDTHH:MM:SSZ) &p3Type=boolean &p3Value=(true|false) (this is to indicate if you want the original deployment to be closed) &p4Type=Date &p4Value=XXXXXX (XXXXXX is the end date for the original deployment (if p3Value is true)) &p5Type=String &p5Value=XXXXXX (XXXXXX is the DataProducer name of the new DataProducer) &p6Type=String &p6Value=XXXXXX (XXXXXX is the base URL to use for the new DataContainers that will be created). &delimiter=| {noformat}\\ Executing the createDuplicateDeepDeployment service call will return an ID for the new deployment. # Edit the new deployment record in the DataProducer table to adjust it's parentID_FK to be for the new mooring. While there edit times and name as appropriate. # Edit the getM? script on tsunami to use the proper device and parent ID (this will be the device ID of the torroid of the mooring deployment) and make sure that the '/oasis/bin/ssdsSubmit.pl $deviceId $parentId "$starttime_es,$endtime_es,$filesize,$rtnsts"' line in the script is configured to run. Here's another example for the October 2010 M1 turn: {noformat} http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&\ p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer|id=35591&p2Type=Date&p2Value=2010-10-27T21:00:00Z&p3Type=boolean&p3Value=false&p4Type=Date&\ p4Value=2010-q0-27T20:30:00Z&p5Type=String&p5Value=getM1-download&p6Type=String&p6Value=&delimiter=| (This returned a new DataProducerID of 41556. The ParentID_FK for this record in the DataProducer table was then changed from 35388 to 41197 and the endDate cleared to null. Also, with the change to alternating deviceIDs implemented in 2010 the deviceID_FK was changed from 1698 to 1768 for this M1 deployment. {noformat} N.B. A rotating scheme for the virtual device IDs was implemented in 2010. This permits the steps described above to be executed prior to the mooring turn during the extensive dock-side test period. h4. Enable processing for other "virtual" devices # Turn on ClockSync processing. We also use a virtual device ID for these data. Simply uncomment the line for it in the ssds.cfg file. # Turn on ISUS processing. Simply uncomment the line for it in the ssds.cfg file. (Device ID re-used from previous deployment.) h4. Make all of the child instrument deployment start times the same as the mooring start time When SSDS receives a packet from an instrument that is not currently deployed it will create a Deployment record (in the DataProducer table) with a startDate that is set to the time of the first record received. As the mooring starts up all of the instrument deployments will have different start times based on when they each first saw data. This can present problems for the data processing that follows, especially for the jobs that aggregate the microcat data into a single ZT file that is used to produce the contour temperature and salinity wind stick plots. To prevent these problems it's best to edit the startDates of the child instrument deployments so that they are all the same. This is currently most easily done through Enterprise Manager with a query like below (this is for the 201010 M1 deployment) to get all the child instrument deployments: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 41197) ORDER BY startDate {noformat} Then copy and paste the datetime string from one cell to the next. The times are GMT. h4. Cycle links to previous deployment # Edit previous.html file in /mbari/ssdsdata/deployments to add a line for the new deployment and add the end date and archive url for the just closed deployment, e.g.: {noformat}
  • 200904 2009-04-29 to 2010-04-03
  • 201004 2010-04-03 to present
    • {noformat}\\ # Add lines to DEPLOYMENTS file in dev/DPforSSDS/cimt/ for the just closed deployment, e.g.: {noformat} DStoNetCDF.pl -mooring M2 -deployment "M2 - April 2009" -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org -outputDir \ /mbari/ssdsdata/deployments -procClosed -verbose combineM.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments combineTS.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments combineAll.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments and /bin/cp /mbari/ssdsdata/deployments/m2/200904/OS_*.nc /mbari/FTP/pub/OceanSITES {noformat}\\ # Execute these lines to generate "closed deployment" data products and web pages and for submission to the OceanSITES GDAC # Confirm that the processing executed properly. Sometimes mangled timestamp data weasels its way into the data stream causing instrument netcdf files to be named with bogus start dates. (This can also happen during a deployment and is one of the maintenance tasks one should follow to keep the data flowing to where it needs to go.) The combine__.pl scripts will then create the OceanSITES formatted files encompassing the dates of all the instrument netcdf files. Incorrect instrument file names will create incorrect OS_MBARI* file names causing CenCOOS and NDBC/Ifremer to complain about the correctly named file not being updated with new data. The fix involves a purging of the bad files, double checking the metadata in SSDS and reprocessing, with perhaps additional checks in the instrument processing perl code to skip over bad records. This is best done at the Unix command line by cd'ing to the deployment directory, e.g. {noformat} cd /mbari/deployments/m1/201010 ls -lrt {noformat} and removing files that have bogus dates that do not represent the deployment start date. You should also remove all the directories and files in the gifs/ subdirectory. These will all get recreated when you run DStoNetCDF.pl and the combine__.pl scripts. With luck, simply removing the bad files and rerunning the processing scripts will fix things. If not, identify where the bad dates are coming from and fix as appropriate. Mike McCann (First edit: 30 October 2007, Last updated: 1 March 2012)]]>     2162834     1933327 1 1835090 2007-10-03 12:37:15.027 2007-10-03 12:37:15.027 1933328 75 1835090 2007-10-03 12:38:15.020 2008-05-02 08:49:15.013 1933333 14 1835090 2007-10-03 13:12:15.027 2007-10-03 13:26:15.023 1933332 1 1835090 2007-10-03 13:12:15.023 2007-10-03 13:12:15.023 1933335 14 1835090 2007-10-03 13:12:15.030 2008-05-02 08:47:15.023 1933336 29 1835090 2007-10-03 13:14:15.017 2008-05-02 08:46:15.023 10356341 10389056 40 2007-10-29 10:49:47.743 2009-05-04 13:58:10.527 2162834 34406583 2064386 2026-02-04 08:34:59.108 2026-02-04 08:34:59.108 34406584 2064386 2026-02-04 08:34:59.108 2026-02-04 08:34:59.108 34406581 2064386 2026-02-04 08:34:59.108 2026-02-04 08:34:59.108 34406582 2064386 2026-02-04 08:34:59.108 2026-02-04 08:34:59.108 2293796 2 1835090 2007-11-06 16:14:15.020 2007-11-06 16:14:15.020 2293797 1 1835140 2007-11-06 16:14:15.043 2007-11-06 16:14:15.043 19300604 2064386 2012-06-18 17:29:23.133 2012-06-18 17:29:23.133 2195625 tm_year + 1900)) + tp->tm_yday + 1; else itime = (1000 * tp->tm_year) + tp->tm_yday + 1;dtime = tp->tm_yday + 1.0 + (double)((3600 * tp->tm_hour) + (60 * tp->tm_min) + tp->tm_sec) / 86400.0;len_got = cc - 9; /* compute amount of log data gotten*/ if ( len_got > 0 ) memmove( buffer, buffer + 9, sizeof(buffer) - 9 ); else if ( len_got < 0 ) /* Move log data to start of buffer*/ { printf("Incomplete record header in %s, record %d (type %d)\n", filename, hdr.log_nmbr,hdr.log_type); continue; } : : : : {noformat} {noformat} {noformat}]]> 2162859 2293794 2 1835090 2007-11-06 16:13:15.023 2008-05-02 08:11:15.027 2293795 1 1835140 2007-11-06 16:14:15.017 2007-11-06 16:14:15.017 2293800 8 2162859 2007-11-06 16:29:15.043 2007-11-07 09:29:15.020 2293801 1 2162859 2007-11-07 09:23:15.120 2007-11-07 09:23:15.120 3707731 3637814 2008-05-02 08:43:44.520 2008-05-02 08:43:44.520 2293798 2 1835090 2007-11-06 16:17:15.017 2007-11-06 16:26:15.027 2293799 1 2162859 2007-11-06 16:29:15.027 2007-11-06 16:29:15.027 26837005 16777265 2011-04-15 06:27:37.837 2014-07-21 15:12:44.687 21450 25 16973825 26837004 16777265 2011-04-15 06:27:37.837 2014-07-21 14:55:06.207 21614 24 16973825 16777269 16810037 3 2011-04-15 06:27:16.287 2011-04-15 07:49:47.603 16777265 26837003 16777265 2011-04-15 06:27:37.837 2011-04-15 11:00:27.190 21723 23 16973825 16777268 16810036 2 2011-04-15 06:27:16.287 2011-04-15 06:27:37.890 16777265 16973855 16777265 2011-04-15 06:27:37.837 2011-04-15 10:57:43.597 21851 21 16973825 16973854 16777265 2011-04-15 06:27:37.837 2011-04-15 10:49:37.647 21733 20 16973825 3637817 3670567 2 2008-05-02 08:15:36.643 2008-05-02 08:20:25.903 3637814 16973857 16777265 2011-04-15 07:53:38.250 2011-04-15 09:41:52.577 39069 9 16973831 16973856 16777265 2011-04-15 06:27:37.837 2011-04-15 10:58:39.607 21959 22 16973825 3637815 3670565 1 2008-05-02 08:15:36.643 2008-05-02 08:15:36.643 3637814 9797946 9830713 35 2007-10-29 10:49:47.743 2009-03-09 15:44:12.127 2162834 16777266 16810034 1 2011-04-15 06:27:16.287 2011-04-15 06:27:16.287 16777265 16777283 16810051 6 2011-04-15 06:27:16.287 2011-04-15 09:39:24.230 16777265 10911787 10944554 43 2007-10-29 10:49:47.743 2009-10-20 09:56:17.793 2162834 3637821 3670571 4 2008-05-02 08:15:36.643 2008-05-02 08:37:28.877 3637814 16777277 16810045 4 2011-04-15 06:27:16.287 2011-04-15 07:53:38.250 16777265 9372126 9404870 32 2007-10-29 10:49:47.743 2009-03-09 09:47:31.360 2162834 3637819 3670569 3 2008-05-02 08:15:36.643 2008-05-02 08:24:18.157 3637814 16777281 16810049 40 2007-10-03 12:37:04.993 2009-04-29 12:37:52.620 1835090 3604513 1 2162834 2008-04-17 08:22:15.067 2008-04-17 08:22:15.067 26837006 16777265 2011-04-15 07:53:38.250 2011-04-15 11:08:06.987 39931 10 16973831 16777279 16810047 5 2011-04-15 06:27:16.287 2011-04-15 09:36:40.353 16777265 3637823 3670573 37 2007-10-03 12:37:04.993 2007-11-06 16:13:13.433 1835090 16973829 16777265 2011-04-15 06:27:37.837 2011-04-15 07:48:30.010 15616 4 16973825 16973828 16777265 2011-04-15 06:27:37.837 2011-04-15 07:33:05.637 12913 3 16973825 16973827 16777265 2011-04-15 06:27:37.837 2011-04-15 07:07:37.473 8131 2 16973825 16973826 16777265 2011-04-15 06:27:37.837 2011-04-15 06:27:37.837 133 1 16973825 16973825 16777265 2011-04-15 06:27:37.837 2014-07-21 15:20:04.863 20765 26 16973837 16777265 2011-04-15 07:53:38.250 2011-04-15 09:28:54.987 38864 6 16973831 16973836 16777265 2011-04-15 07:53:38.250 2011-04-15 09:28:46.660 38864 5 16973831 16973835 16777265 2011-04-15 07:53:38.250 2011-04-15 08:27:31.127 14667 4 16973831 16973834 16777265 2011-04-15 07:53:38.250 2011-04-15 08:25:22.937 13696 3 16973831 16973833 16777265 2011-04-15 07:53:38.250 2011-04-15 08:11:56.917 6922 2 16973831 16973832 16777265 2011-04-15 07:53:38.250 2011-04-15 07:53:38.250 133 1 16973831 16973831 16777265 2011-04-15 07:53:38.250 2014-07-21 15:22:41.117 39543 11 9797971 9830737 36 2007-10-29 10:49:47.743 2009-04-29 15:21:07.837 2162834 16973830 16777265 2011-04-15 06:27:37.837 2011-04-15 07:50:45.037 15617 5 16973825 16973844 16777265 2011-04-15 06:27:37.837 2011-04-15 10:13:01.487 18404 10 16973825 16973845 16777265 2011-04-15 06:27:37.837 2011-04-15 10:14:26.427 18413 11 16973825 16973842 16777265 2011-04-15 06:27:37.837 2011-04-15 10:06:20.957 18323 8 16973825 16973843 16777265 2011-04-15 06:27:37.837 2011-04-15 10:09:28.840 18322 9 16973825 16973840 16777265 2011-04-15 06:27:37.837 2011-04-15 07:51:25.637 15615 6 16973825 16973841 16777265 2011-04-15 06:27:37.837 2011-04-15 10:03:58.173 18297 7 16973825 16973838 16777265 2011-04-15 07:53:38.250 2011-04-15 09:31:09.460 38857 7 16973831 9372106 9404851 31 2007-10-29 10:49:47.743 2008-11-24 10:05:07.423 2162834 16973839 16777265 2011-04-15 07:53:38.250 2011-04-15 09:40:45.637 38863 8 16973831 16973852 16777265 2011-04-15 06:27:37.837 2011-04-15 10:44:23.227 21737 18 16973825 16973853 16777265 2011-04-15 06:27:37.837 2011-04-15 10:44:47.110 21740 19 16973825 16973850 16777265 2011-04-15 06:27:37.837 2011-04-15 10:28:11.077 19747 16 16973825 16973851 16777265 2011-04-15 06:27:37.837 2011-04-15 10:40:34.223 21672 17 16973825 16973848 16777265 2011-04-15 06:27:37.837 2011-04-15 10:23:22.243 18463 14 16973825 16973849 16777265 2011-04-15 06:27:37.837 2011-04-15 10:25:18.437 18478 15 16973825 16973846 16777265 2011-04-15 06:27:37.837 2011-04-15 10:15:05.750 18405 12 16973825 16973847 16777265 2011-04-15 06:27:37.837 2011-04-15 10:18:43.920 18454 13 16973825 4849685 1 3637814 2008-06-03 13:15:15.017 2008-06-03 13:15:15.017 18579804 18645318 58 2007-10-03 12:37:04.993 2012-02-03 16:39:09.343 1835090 18579805 18645319 59 2007-10-03 12:37:04.993 2012-02-03 16:40:12.230 1835090 18579806 18645320 3 2011-07-07 20:40:04.910 2011-07-07 20:52:18.727 17858815 18579807 18645321 60 2007-10-03 12:37:04.993 2012-02-03 16:40:27.637 1835090 18579808 18645322 61 2007-10-03 12:37:04.993 2012-02-03 16:41:04.237 1835090 18579809 18645323 4 2011-07-07 20:40:04.910 2012-02-03 16:40:27.650 17858815 18579788 18645303 54 2007-10-03 12:37:04.993 2012-02-03 09:26:52.010 1835090 18579790 18645305 55 2007-10-03 12:37:04.993 2012-02-03 09:28:45.737 1835090 18579791 18645306 56 2007-10-03 12:37:04.993 2012-02-03 09:44:15.057 1835090 2162926 2195692 14 2007-10-29 10:49:47.743 2007-11-16 08:41:58.810 2162834 2162925 2195691 13 2007-10-29 10:49:47.743 2007-11-09 15:42:05.530 2162834 18579797 18645312 1 2012-02-03 12:04:08.647 2012-02-03 12:04:08.647 18579795 9797896 9830664 39 2007-10-03 12:37:04.993 2009-04-29 12:37:23.333 1835090 2162929 2195695 16 2007-10-29 10:49:47.743 2007-11-16 11:52:15.517 2162834 9797895 9830663 38 2007-10-03 12:37:04.993 2008-05-02 08:46:05.477 1835090 2162928 2195694 15 2007-10-29 10:49:47.743 2007-11-16 10:21:10.833 2162834 18579802 18645316 57 2007-10-03 12:37:04.993 2012-02-03 11:55:21.913 1835090 18579801 18645315 2 2008-04-02 11:57:23.537 2012-02-03 16:38:46.150 3114486 6259316 6226519 27 2007-10-29 10:49:47.743 2008-09-30 12:53:19.247 2162834 3604633 2 1835140 2008-05-02 08:14:15.027 2008-05-02 08:14:15.027 6259318 6226521 28 2007-10-29 10:49:47.743 2008-10-08 13:19:15.573 2162834 18579771 18645286 67 2007-10-29 10:49:47.743 2012-02-03 09:16:05.400 2162834 6259320 6226523 29 2007-10-29 10:49:47.743 2008-10-08 14:30:26.127 2162834 18579777 18645292 46 2007-10-03 12:37:04.993 2012-02-03 09:17:01.527 1835090 11240445 11273182 49 2007-10-29 10:49:47.743 2010-03-16 21:28:47.483 2162834 18579778 18645293 47 2007-10-03 12:37:04.993 2012-02-03 09:20:42.827 1835090 18579775 18645290 45 2007-10-03 12:37:04.993 2012-02-03 09:15:18.437 1835090 11240443 11273180 48 2007-10-29 10:49:47.743 2010-03-16 21:27:25.397 2162834 18579782 18645297 50 2007-10-03 12:37:04.993 2012-02-03 09:23:13.327 1835090 18579779 18645294 48 2007-10-03 12:37:04.993 2012-02-03 09:20:49.303 1835090 18579780 18645295 49 2007-10-03 12:37:04.993 2012-02-03 09:21:11.363 1835090 18579785 18645300 52 2007-10-03 12:37:04.993 2012-02-03 09:25:50.710 1835090 10911785 10944552 42 2007-10-29 10:49:47.743 2009-10-20 08:16:56.887 2162834 18579786 18645301 53 2007-10-03 12:37:04.993 2012-02-03 09:26:14.190 1835090 10911784 10944551 41 2007-10-29 10:49:47.743 2009-09-30 11:07:38.320 2162834 18579783 18645298 51 2007-10-03 12:37:04.993 2012-02-03 09:25:09.793 1835090 18579784 18645299 12 2007-10-05 14:24:44.267 2007-11-06 16:24:38.987 1835140 11240442 11273179 47 2007-10-29 10:49:47.743 2010-03-05 10:18:00.237 2162834 18579770 18645285 44 2007-10-03 12:37:04.993 2012-02-03 09:13:53.413 1835090 18579768 18645283 43 2007-10-03 12:37:04.993 2011-07-07 20:36:45.507 1835090 9830908 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs to the newly deployed deviceIDs (aka ISI_IDs). Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml wile with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /hosts/tornado_vol0/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed. h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSD ingest will recognize them as new. # +Do not close (set endDates) the existing Test deployment until after the oasisToSSDS has run with the new platform deployment name+ and all new instrument deployments have been created in SSDS_Metadata. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadta, e.g.: {noformat} SELECT id, name, startDate, endDate, ParentID_FK FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') ORDER BY id DESC {noformat} Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file. # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. # When the mooring log message is sent to oasis with the actual time of deployment enter that as the startDate in the database.  All times (DTGs) in the SSDS_Metadata database are GMT. Make sure to set the startDate for the imctd microcat sensors too. This can be done with a query like this where you use the proper dataProduceIDs for the mooring and the imcd: {noformat} SELECT id, name, ParentID_FK, startDate, endDate, nominalDepth FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') AND (ParentID_FK = 31564) OR (ParentID_FK = 31548) ORDER BY id DESC {noformat} Mike McCann (First edit: 30 October 2007, Last updated: 1 May 2009)]]> 9798154 18645467 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs and the path to the xml files to the newly deployed deviceIDs (aka ISI_IDs) and new xml directory. Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen or JEdit. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml file with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. \[Note: With the M1 - October 2009 deployment we added the individual inductive modem microcats as children of the mooring and also configured a parallel deployment of the TString that produced the same data. With the M2 - April 2010 deployment we configured the individual IM mirocrocats and did not configure a parallel TString deployment. This is a better model for the system and is much easier to configure and produces more easily consumed data by downstream processes such as combineTS.pl.\] With future deployments we will not configure TString. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|SSDS:Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[SSDS:main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /mbari/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. \\ For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed (or...as below \-rschramm 4/2010) \\ \\ {noformat} declare @myID as bigint set @myID = 31548 SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE id = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk in ( SELECT id FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL) ORDER BY id {noformat} h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSDS ingest will recognize them as new. # +Do not close (set endDates) the existing Test deployment until after the oasisToSSDS has run with the new platform deployment name+ and all new instrument deployments have been created in SSDS_Metadata. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadata, e.g.: {noformat} SELECT id, name, startDate, endDate, ParentID_FK FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') ORDER BY id DESC {noformat} Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file. # You will also need to edit the DStoNetCDF.pl script to set the new name of the M1 deployment, e.g.: {noformat} # # SSDS mooring name Deployment lookup # my %ssdsMooringDeplNames = ( M0 => 'M0', M1 => 'M1 - October 2009', M1Test => 'Test M1 - October 2009', M2 => 'M2 - April 2009', {noformat} # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. # When the mooring log message is sent to oasis with the actual time of deployment enter that as the startDate in the database.  All times (DTGs) in the SSDS_Metadata database are GMT. Make sure to set the startDate for the imctd microcat sensors too. This can be done with a query like this where you use the proper dataProduceIDs for the mooring and the imcd: {noformat} SELECT id, name, ParentID_FK, startDate, endDate, nominalDepth FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') AND (ParentID_FK = 31564) OR (ParentID_FK = 31548) ORDER BY id DESC {noformat} # Because of some bug in SSDS ingest the dataContainerTypes of the outputs from the instrument deployments do not get properly assigned the values of 'Stream'.  This needs to be fixed so that the NDBC datatransfers will work. To fix it edit the SSDS_Metadata database starting with a query like this: {noformat} SELECT ssdsdba.DataContainer.* FROM ssdsdba.DataProducer INNER JOIN ssdsdba.DataContainer ON ssdsdba.DataProducer.id = ssdsdba.DataContainer.DataProducerID_FK WHERE (ssdsdba.DataProducer.ParentID_FK = 31548) {noformat} where you use the DataProducerID for the new mooring deployment in the WHERE clause. ==> Change all of the 'File's in the dataContainerType' field to 'Stream's. h2. C. Procedures to be done after the mooring turn h4. Set up download info deployment and reporting Though not a real instrument, we configure a virtual 'dlinfo' instrument for the download scripts to attach download statistics data. We re-use the same device IDs for the M1 and M2 moorings, so it's best to configure this after the new mooring is out and all of those deployments have been closed. As the data are delivered "out of band" from the typical OASIS instruments we need to create a deployment for dlinfo instrument by hand. # The most direct way is to use the createDuplicateDeepDeployment service call. For example, to duplicate the 2009 M2 dlinfo deployment for the 2010 M2 deployment this call was executed: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer\|id=*34496*&p2Type=Date&p2Value=*2010-04-03T22:00:00Z*&p3Type=boolean&p3Value=false&p4Type=Date&p4Value=*2010-04-03T23:00:00Z*&p5Type=String&p5Value=getM2-download&p6Type=String&p6Value=&delimiter=\|. Of course you will need to adjust the DataProducer ID Date values for the new deployment you are creating. Here is the Key to the parameters: {noformat} Key: ---- http://localhost:8080/servlet/MetadataAccessServlet ?responseType=text &delimiter=| &objectToInvokeOn=DataProducerAccess &method=createDuplicateDeepDeployment &p1Type=DataProducer &p1Value=DataProducer|id=XXXX (XXXX is the ID of the deployment to copy) &p2Type=Date &p2Value=XXXXXX (XXXXXX is the start date of the new copy in XML format YYYY-MM-DDTHH:MM:SSZ) &p3Type=boolean &p3Value=(true|false) (this is to indicate if you want the original deployment to be closed) &p4Type=Date &p4Value=XXXXXX (XXXXXX is the end date for the original deployment (if p3Value is true)) &p5Type=String &p5Value=XXXXXX (XXXXXX is the DataProducer name of the new DataProducer) &p6Type=String &p6Value=XXXXXX (XXXXXX is the base URL to use for the new DataContainers that will be created). &delimiter=| {noformat}\\ Executing the createDuplicateDeepDeployment service call will return an ID for the new deployment. # Edit the new deployment record in the DataProducer table to adjust it's parentID_FK to be for the new mooring. While there edit times and name as appropriate. # Edit the getM? script on tsunami to use the proper device and parent ID (this will be the device ID of the torroid of the mooring deployment) and make sure that the '/oasis/bin/ssdsSubmit.pl $deviceId $parentId "$starttime_es,$endtime_es,$filesize,$rtnsts"' line in the script is configured to run. Here's another example for the October 2010 M1 turn: {noformat} http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&\ p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer|id=35591&p2Type=Date&p2Value=2010-10-27T21:00:00Z&p3Type=boolean&p3Value=false&p4Type=Date&\ p4Value=2010-q0-27T20:30:00Z&p5Type=String&p5Value=getM1-download&p6Type=String&p6Value=&delimiter=| (This returned a new DataProducerID of 41556. The ParentID_FK for this record in the DataProducer table was then changed from 35388 to 41197 and the endDate cleared to null. Also, with the change to alternating deviceIDs implemented in 2010 the deviceID_FK was changed from 1698 to 1768 for this M1 deployment. {noformat} N.B. A rotating scheme for the virtual device IDs was implemented in 2010. This permits the steps described above to be executed prior to the mooring turn during the extensive dock-side test period. h4. Enable processing for other "virtual" devices # Turn on ClockSync processing. We also use a virtual device ID for these data. Simply uncomment the line for it in the ssds.cfg file. # Turn on ISUS processing. Simply uncomment the line for it in the ssds.cfg file. (Device ID re-used from previous deployment.) h4. Make all of the child instrument deployment start times the same as the mooring start time When SSDS receives a packet from an instrument that is not currently deployed it will create a Deployment record (in the DataProducer table) with a startDate that is set to the time of the first record received. As the mooring starts up all of the instrument deployments will have different start times based on when they each first saw data. This can present problems for the data processing that follows, especially for the jobs that aggregate the microcat data into a single ZT file that is used to produce the contour temperature and salinity wind stick plots. To prevent these problems it's best to edit the startDates of the child instrument deployments so that they are all the same. This is currently most easily done through Enterprise Manager with a query like below (this is for the 201010 M1 deployment) to get all the child instrument deployments: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 41197) ORDER BY startDate {noformat} Then copy and paste the datetime string from one cell to the next. The times are GMT. h4. Cycle links to previous deployment # Edit previous.html file in /mbari/ssdsdata/deployments to add a line for the new deployment and add the end date and archive url for the just closed deployment, e.g.: {noformat}
  • 200904 2009-04-29 to 2010-04-03
  • 201004 2010-04-03 to present
    • {noformat}\\ # Add lines to DEPLOYMENTS file in dev/DPforSSDS/cimt/ for the just closed deployment, e.g.: {noformat} DStoNetCDF.pl -mooring M2 -deployment "M2 - April 2009" -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org -outputDir \ /mbari/ssdsdata/deployments -procClosed -verbose combineM.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments combineTS.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments combineAll.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments and /bin/cp /mbari/ssdsdata/deployments/m2/200904/OS_*.nc /mbari/FTP/pub/OceanSITES {noformat}\\ # Execute these lines to generate "closed deployment" data products and web pages and for submission to the OceanSITES GDAC Mike McCann (First edit: 30 October 2007, Last updated: 1 May 2009)]]>     18579959     18645470 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs and the path to the xml files to the newly deployed deviceIDs (aka ISI_IDs) and new xml directory. Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen or JEdit. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml file with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. \[Note: With the M1 - October 2009 deployment we added the individual inductive modem microcats as children of the mooring and also configured a parallel deployment of the TString that produced the same data. With the M2 - April 2010 deployment we configured the individual IM mirocrocats and did not configure a parallel TString deployment. This is a better model for the system and is much easier to configure and produces more easily consumed data by downstream processes such as combineTS.pl.\] With future deployments we will not configure TString. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|SSDS:Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[SSDS:main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /mbari/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. \\ For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed (or...as below \-rschramm 4/2010) \\ \\ {noformat} declare @myID as bigint set @myID = 31548 SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE id = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk in ( SELECT id FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL) ORDER BY id {noformat} h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSDS ingest will recognize them as new. # +Do not close (set endDates) the existing Test deployment until after the oasisToSSDS has run with the new platform deployment name+ and all new instrument deployments have been created in SSDS_Metadata. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadata, e.g.: {noformat} SELECT id, name, startDate, endDate, ParentID_FK FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') ORDER BY id DESC {noformat} Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file. # You will also need to edit the DStoNetCDF.pl script to set the new name of the M1 deployment, e.g.: {noformat} # # SSDS mooring name Deployment lookup # my %ssdsMooringDeplNames = ( M0 => 'M0', M1 => 'M1 - October 2009', M1Test => 'Test M1 - October 2009', M2 => 'M2 - April 2009', {noformat} # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. # When the mooring log message is sent to oasis with the actual time of deployment enter that as the startDate in the database.  All times (DTGs) in the SSDS_Metadata database are GMT. Make sure to set the startDate for the imctd microcat sensors too. This can be done with a query like this where you use the proper dataProduceIDs for the mooring and the imcd: {noformat} SELECT id, name, ParentID_FK, startDate, endDate, nominalDepth FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') AND (ParentID_FK = 31564) OR (ParentID_FK = 31548) ORDER BY id DESC {noformat} # Because of some bug in SSDS ingest the dataContainerTypes of the outputs from the instrument deployments do not get properly assigned the values of 'Stream'.  This needs to be fixed so that the NDBC datatransfers will work. To fix it edit the SSDS_Metadata database starting with a query like this: {noformat} SELECT ssdsdba.DataContainer.* FROM ssdsdba.DataProducer INNER JOIN ssdsdba.DataContainer ON ssdsdba.DataProducer.id = ssdsdba.DataContainer.DataProducerID_FK WHERE (ssdsdba.DataProducer.ParentID_FK = 31548) {noformat} where you use the DataProducerID for the new mooring deployment in the WHERE clause. ==> Change all of the 'File's in the dataContainerType' field to 'Stream's. h2. C. Procedures to be done after the mooring turn h4. Set up download info deployment and reporting Though not a real instrument, we configure a virtual 'dlinfo' instrument for the download scripts to attach download statistics data. We re-use the same device IDs for the M1 and M2 moorings, so it's best to configure this after the new mooring is out and all of those deployments have been closed. As the data are delivered "out of band" from the typical OASIS instruments we need to create a deployment for dlinfo instrument by hand. # The most direct way is to use the createDuplicateDeepDeployment service call. For example, to duplicate the 2009 M2 dlinfo deployment for the 2010 M2 deployment this call was executed: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer\|id=*34496*&p2Type=Date&p2Value=*2010-04-03T22:00:00Z*&p3Type=boolean&p3Value=false&p4Type=Date&p4Value=*2010-04-03T23:00:00Z*&p5Type=String&p5Value=getM2-download&p6Type=String&p6Value=&delimiter=\|. Of course you will need to adjust the DataProducer ID Date values for the new deployment you are creating. Here is the Key to the parameters: {noformat} Key: ---- http://localhost:8080/servlet/MetadataAccessServlet ?responseType=text &delimiter=| &objectToInvokeOn=DataProducerAccess &method=createDuplicateDeepDeployment &p1Type=DataProducer &p1Value=DataProducer|id=XXXX (XXXX is the ID of the deployment to copy) &p2Type=Date &p2Value=XXXXXX (XXXXXX is the start date of the new copy in XML format YYYY-MM-DDTHH:MM:SSZ) &p3Type=boolean &p3Value=(true|false) (this is to indicate if you want the original deployment to be closed) &p4Type=Date &p4Value=XXXXXX (XXXXXX is the end date for the original deployment (if p3Value is true)) &p5Type=String &p5Value=XXXXXX (XXXXXX is the DataProducer name of the new DataProducer) &p6Type=String &p6Value=XXXXXX (XXXXXX is the base URL to use for the new DataContainers that will be created). &delimiter=| {noformat}\\ Executing the createDuplicateDeepDeployment service call will return an ID for the new deployment. # Edit the new deployment record in the DataProducer table to adjust it's parentID_FK to be for the new mooring. While there edit times and name as appropriate. # Edit the getM? script on tsunami to use the proper device and parent ID (this will be the device ID of the torroid of the mooring deployment) and make sure that the '/oasis/bin/ssdsSubmit.pl $deviceId $parentId "$starttime_es,$endtime_es,$filesize,$rtnsts"' line in the script is configured to run. Here's another example for the October 2010 M1 turn: {noformat} http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&\ p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer|id=35591&p2Type=Date&p2Value=2010-10-27T21:00:00Z&p3Type=boolean&p3Value=false&p4Type=Date&\ p4Value=2010-q0-27T20:30:00Z&p5Type=String&p5Value=getM1-download&p6Type=String&p6Value=&delimiter=| (This returned a new DataProducerID of 41556. The ParentID_FK for this record in the DataProducer table was then changed from 35388 to 41197 and the endDate cleared to null. Also, with the change to alternating deviceIDs implemented in 2010 the deviceID_FK was changed from 1698 to 1768 for this M1 deployment. {noformat} N.B. A rotating scheme for the virtual device IDs was implemented in 2010. This permits the steps described above to be executed prior to the mooring turn during the extensive dock-side test period. h4. Enable processing for other "virtual" devices # Turn on ClockSync processing. We also use a virtual device ID for these data. Simply uncomment the line for it in the ssds.cfg file. # Turn on ISUS processing. Simply uncomment the line for it in the ssds.cfg file. (Device ID re-used from previous deployment.) h4. Make all of the child instrument deployment start times the same as the mooring start time When SSDS receives a packet from an instrument that is not currently deployed it will create a Deployment record (in the DataProducer table) with a startDate that is set to the time of the first record received. As the mooring starts up all of the instrument deployments will have different start times based on when they each first saw data. This can present problems for the data processing that follows, especially for the jobs that aggregate the microcat data into a single ZT file that is used to produce the contour temperature and salinity wind stick plots. To prevent these problems it's best to edit the startDates of the child instrument deployments so that they are all the same. This is currently most easily done through Enterprise Manager with a query like below (this is for the 201010 M1 deployment) to get all the child instrument deployments: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 41197) ORDER BY startDate {noformat} Then copy and paste the datetime string from one cell to the next. The times are GMT. h4. Cycle links to previous deployment # Edit previous.html file in /mbari/ssdsdata/deployments to add a line for the new deployment and add the end date and archive url for the just closed deployment, e.g.: {noformat}
  • 200904 2009-04-29 to 2010-04-03
  • 201004 2010-04-03 to present
    • {noformat}\\ # Add lines to DEPLOYMENTS file in dev/DPforSSDS/cimt/ for the just closed deployment, e.g.: {noformat} DStoNetCDF.pl -mooring M2 -deployment "M2 - April 2009" -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org -outputDir \ /mbari/ssdsdata/deployments -procClosed -verbose combineM.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments combineTS.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments combineAll.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments and /bin/cp /mbari/ssdsdata/deployments/m2/200904/OS_*.nc /mbari/FTP/pub/OceanSITES {noformat}\\ # Execute these lines to generate "closed deployment" data products and web pages and for submission to the OceanSITES GDAC # Confirm that the processing executed properly. Sometimes mangled timestamp data weasels its way into the data stream causing instrument netcdf files to be named with bogus start dates. (This can also happen during a deployment and is one of the maintenance tasks one should follow to keep the data flowing to where it needs to go.) The combine__.pl scripts will then create the OceanSITES formatted files encompassing the dates of all the instrument netcdf files. Incorrect instrument file names will create incorrect OS_MBARI* file names causing CenCOOS and NDBC/Ifremer to complain about the correctly named file not being updated with new data. The fix involves a purging of the bad files, double checking the metadata in SSDS and reprocessing, with perhaps additional checks in the instrument processing perl code to skip over bad records. This is best done at the Unix command line by cd'ing to the deployment directory, e.g. {noformat} cd /mbari/deployments/m1/201010 ls -lrt {noformat} and removing files that have bogus dates that do not represent the deployment start date. You should also remove all the files directories and files in the gifs/ subdirectory. These will all get recreated when you run DStoNetCDF.pl and the combine__.pl scripts. With luck simply removing the bad files and rerunning the processing scripts will fix things. If not, identify where the bad dates are coming from and fix as appropriate. Mike McCann (First edit: 30 October 2007, Last updated: 1 May 2009)]]>     18579962     18645469 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs and the path to the xml files to the newly deployed deviceIDs (aka ISI_IDs) and new xml directory. Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen or JEdit. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml file with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. \[Note: With the M1 - October 2009 deployment we added the individual inductive modem microcats as children of the mooring and also configured a parallel deployment of the TString that produced the same data. With the M2 - April 2010 deployment we configured the individual IM mirocrocats and did not configure a parallel TString deployment. This is a better model for the system and is much easier to configure and produces more easily consumed data by downstream processes such as combineTS.pl.\] With future deployments we will not configure TString. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|SSDS:Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[SSDS:main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /mbari/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. \\ For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed (or...as below \-rschramm 4/2010) \\ \\ {noformat} declare @myID as bigint set @myID = 31548 SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE id = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk in ( SELECT id FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL) ORDER BY id {noformat} h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSDS ingest will recognize them as new. # +Do not close (set endDates) the existing Test deployment until after the oasisToSSDS has run with the new platform deployment name+ and all new instrument deployments have been created in SSDS_Metadata. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadata, e.g.: {noformat} SELECT id, name, startDate, endDate, ParentID_FK FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') ORDER BY id DESC {noformat} Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file. # You will also need to edit the DStoNetCDF.pl script to set the new name of the M1 deployment, e.g.: {noformat} # # SSDS mooring name Deployment lookup # my %ssdsMooringDeplNames = ( M0 => 'M0', M1 => 'M1 - October 2009', M1Test => 'Test M1 - October 2009', M2 => 'M2 - April 2009', {noformat} # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. # When the mooring log message is sent to oasis with the actual time of deployment enter that as the startDate in the database.  All times (DTGs) in the SSDS_Metadata database are GMT. Make sure to set the startDate for the imctd microcat sensors too. This can be done with a query like this where you use the proper dataProduceIDs for the mooring and the imcd: {noformat} SELECT id, name, ParentID_FK, startDate, endDate, nominalDepth FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') AND (ParentID_FK = 31564) OR (ParentID_FK = 31548) ORDER BY id DESC {noformat} # Because of some bug in SSDS ingest the dataContainerTypes of the outputs from the instrument deployments do not get properly assigned the values of 'Stream'.  This needs to be fixed so that the NDBC datatransfers will work. To fix it edit the SSDS_Metadata database starting with a query like this: {noformat} SELECT ssdsdba.DataContainer.* FROM ssdsdba.DataProducer INNER JOIN ssdsdba.DataContainer ON ssdsdba.DataProducer.id = ssdsdba.DataContainer.DataProducerID_FK WHERE (ssdsdba.DataProducer.ParentID_FK = 31548) {noformat} where you use the DataProducerID for the new mooring deployment in the WHERE clause. ==> Change all of the 'File's in the dataContainerType' field to 'Stream's. h2. C. Procedures to be done after the mooring turn h4. Set up download info deployment and reporting Though not a real instrument, we configure a virtual 'dlinfo' instrument for the download scripts to attach download statistics data. We re-use the same device IDs for the M1 and M2 moorings, so it's best to configure this after the new mooring is out and all of those deployments have been closed. As the data are delivered "out of band" from the typical OASIS instruments we need to create a deployment for dlinfo instrument by hand. # The most direct way is to use the createDuplicateDeepDeployment service call. For example, to duplicate the 2009 M2 dlinfo deployment for the 2010 M2 deployment this call was executed: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer\|id=*34496*&p2Type=Date&p2Value=*2010-04-03T22:00:00Z*&p3Type=boolean&p3Value=false&p4Type=Date&p4Value=*2010-04-03T23:00:00Z*&p5Type=String&p5Value=getM2-download&p6Type=String&p6Value=&delimiter=\|. Of course you will need to adjust the DataProducer ID Date values for the new deployment you are creating. Here is the Key to the parameters: {noformat} Key: ---- http://localhost:8080/servlet/MetadataAccessServlet ?responseType=text &delimiter=| &objectToInvokeOn=DataProducerAccess &method=createDuplicateDeepDeployment &p1Type=DataProducer &p1Value=DataProducer|id=XXXX (XXXX is the ID of the deployment to copy) &p2Type=Date &p2Value=XXXXXX (XXXXXX is the start date of the new copy in XML format YYYY-MM-DDTHH:MM:SSZ) &p3Type=boolean &p3Value=(true|false) (this is to indicate if you want the original deployment to be closed) &p4Type=Date &p4Value=XXXXXX (XXXXXX is the end date for the original deployment (if p3Value is true)) &p5Type=String &p5Value=XXXXXX (XXXXXX is the DataProducer name of the new DataProducer) &p6Type=String &p6Value=XXXXXX (XXXXXX is the base URL to use for the new DataContainers that will be created). &delimiter=| {noformat}\\ Executing the createDuplicateDeepDeployment service call will return an ID for the new deployment. # Edit the new deployment record in the DataProducer table to adjust it's parentID_FK to be for the new mooring. While there edit times and name as appropriate. # Edit the getM? script on tsunami to use the proper device and parent ID (this will be the device ID of the torroid of the mooring deployment) and make sure that the '/oasis/bin/ssdsSubmit.pl $deviceId $parentId "$starttime_es,$endtime_es,$filesize,$rtnsts"' line in the script is configured to run. Here's another example for the October 2010 M1 turn: {noformat} http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&\ p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer|id=35591&p2Type=Date&p2Value=2010-10-27T21:00:00Z&p3Type=boolean&p3Value=false&p4Type=Date&\ p4Value=2010-q0-27T20:30:00Z&p5Type=String&p5Value=getM1-download&p6Type=String&p6Value=&delimiter=| (This returned a new DataProducerID of 41556. The ParentID_FK for this record in the DataProducer table was then changed from 35388 to 41197 and the endDate cleared to null. Also, with the change to alternating deviceIDs implemented in 2010 the deviceID_FK was changed from 1698 to 1768 for this M1 deployment. {noformat} N.B. A rotating scheme for the virtual device IDs was implemented in 2010. This permits the steps described above to be executed prior to the mooring turn during the extensive dock-side test period. h4. Enable processing for other "virtual" devices # Turn on ClockSync processing. We also use a virtual device ID for these data. Simply uncomment the line for it in the ssds.cfg file. # Turn on ISUS processing. Simply uncomment the line for it in the ssds.cfg file. (Device ID re-used from previous deployment.) h4. Make all of the child instrument deployment start times the same as the mooring start time When SSDS receives a packet from an instrument that is not currently deployed it will create a Deployment record (in the DataProducer table) with a startDate that is set to the time of the first record received. As the mooring starts up all of the instrument deployments will have different start times based on when they each first saw data. This can present problems for the data processing that follows, especially for the jobs that aggregate the microcat data into a single ZT file that is used to produce the contour temperature and salinity wind stick plots. To prevent these problems it's best to edit the startDates of the child instrument deployments so that they are all the same. This is currently most easily done through Enterprise Manager with a query like below (this is for the 201010 M1 deployment) to get all the child instrument deployments: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 41197) ORDER BY startDate {noformat} Then copy and paste the datetime string from one cell to the next. The times are GMT. h4. Cycle links to previous deployment # Edit previous.html file in /mbari/ssdsdata/deployments to add a line for the new deployment and add the end date and archive url for the just closed deployment, e.g.: {noformat}
  • 200904 2009-04-29 to 2010-04-03
  • 201004 2010-04-03 to present
    • {noformat}\\ # Add lines to DEPLOYMENTS file in dev/DPforSSDS/cimt/ for the just closed deployment, e.g.: {noformat} DStoNetCDF.pl -mooring M2 -deployment "M2 - April 2009" -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org -outputDir \ /mbari/ssdsdata/deployments -procClosed -verbose combineM.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments combineTS.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments combineAll.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments and /bin/cp /mbari/ssdsdata/deployments/m2/200904/OS_*.nc /mbari/FTP/pub/OceanSITES {noformat}\\ # Execute these lines to generate "closed deployment" data products and web pages and for submission to the OceanSITES GDAC # Confirm that the processing executed properly. Sometimes mangled timestamp data weasels its way into the data stream causing instrument netcdf files to be named with bogus start dates. (This can also happen during a deployment and is one of the maintenance tasks one should follow to keep the data flowing to where it needs to go.) The combine__.pl scripts will then create the OceanSITES formatted files encompassing the dates of all the instrument netcdf files. Incorrect instrument file names will create incorrect OS_MBARI* file names causing CenCOOS and NDBC/Ifremer to complain about the correctly named file not being updated with new data. The fix involves a purging of the bad files, double checking the metadata in SSDS and reprocessing, with perhaps additional checks in the instrument processing perl code to skip over bad records. This is best done at the Unix command line by cd'ing to the deployment directory, e.g. {noformat} cd /mbari/deployments/m1/201010 ls -lrt {noformat} Mike McCann (First edit: 30 October 2007, Last updated: 1 May 2009)]]>     18579961     16810034 16777266 11273182 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs and the path to the xml files to the newly deployed deviceIDs (aka ISI_IDs) and new xml directory. Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen or JEdit. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml file with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /mbari/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed. h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSDS ingest will recognize them as new. # +Do not close (set endDates) the existing Test deployment until after the oasisToSSDS has run with the new platform deployment name+ and all new instrument deployments have been created in SSDS_Metadata. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadata, e.g.: {noformat} SELECT id, name, startDate, endDate, ParentID_FK FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') ORDER BY id DESC {noformat} Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file. # You will also need to edit the DStoNetCDF.pl script to set the new name of the M1 deployment, e.g.: {noformat} # # SSDS mooring name Deployment lookup # my %ssdsMooringDeplNames = ( M0 => 'M0', M1 => 'M1 - October 2009', M1Test => 'Test M1 - October 2009', M2 => 'M2 - April 2009', {noformat} # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. # When the mooring log message is sent to oasis with the actual time of deployment enter that as the startDate in the database.  All times (DTGs) in the SSDS_Metadata database are GMT. Make sure to set the startDate for the imctd microcat sensors too. This can be done with a query like this where you use the proper dataProduceIDs for the mooring and the imcd: {noformat} SELECT id, name, ParentID_FK, startDate, endDate, nominalDepth FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') AND (ParentID_FK = 31564) OR (ParentID_FK = 31548) ORDER BY id DESC {noformat} # Because of some bug in SSDS ingest the dataContainerTypes of the outputs from the instrument deployments do not get properly assigned the values of 'Stream'.  This needs to be fixed so that the NDBC datatransfers will work. To fix it edit the SSDS_Metadata database starting with a query like this: {noformat} SELECT ssdsdba.DataContainer.* FROM ssdsdba.DataProducer INNER JOIN ssdsdba.DataContainer ON ssdsdba.DataProducer.id = ssdsdba.DataContainer.DataProducerID_FK WHERE (ssdsdba.DataProducer.ParentID_FK = 31548) {noformat} where you use the DataProducerID for the new mooring deployment in the WHERE clause. ==> Change all of the 'File's in the dataContainerType' field to 'Stream's. Mike McCann (First edit: 30 October 2007, Last updated: 1 May 2009)]]> 11240445 11273180 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs and the path to the xml files to the newly deployed deviceIDs (aka ISI_IDs) and new xml directory. Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen or JEdit. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml wile with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /mbari/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed. h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSDS ingest will recognize them as new. # +Do not close (set endDates) the existing Test deployment until after the oasisToSSDS has run with the new platform deployment name+ and all new instrument deployments have been created in SSDS_Metadata. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadata, e.g.: {noformat} SELECT id, name, startDate, endDate, ParentID_FK FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') ORDER BY id DESC {noformat} Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file. # You will also need to edit the DStoNetCDF.pl script to set the new name of the M1 deployment, e.g.: {noformat} # # SSDS mooring name Deployment lookup # my %ssdsMooringDeplNames = ( M0 => 'M0', M1 => 'M1 - October 2009', M1Test => 'Test M1 - October 2009', M2 => 'M2 - April 2009', {noformat} # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. # When the mooring log message is sent to oasis with the actual time of deployment enter that as the startDate in the database.  All times (DTGs) in the SSDS_Metadata database are GMT. Make sure to set the startDate for the imctd microcat sensors too. This can be done with a query like this where you use the proper dataProduceIDs for the mooring and the imcd: {noformat} SELECT id, name, ParentID_FK, startDate, endDate, nominalDepth FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') AND (ParentID_FK = 31564) OR (ParentID_FK = 31548) ORDER BY id DESC {noformat} # Because of some bug in SSDS ingest the dataContainerTypes of the outputs from the instrument deployments do not get properly assigned the values of 'Stream'.  This needs to be fixed so that the NDBC datatransfers will work. To fix it edit the SSDS_Metadata database starting with a query like this: {noformat} SELECT ssdsdba.DataContainer.* FROM ssdsdba.DataProducer INNER JOIN ssdsdba.DataContainer ON ssdsdba.DataProducer.id = ssdsdba.DataContainer.DataProducerID_FK WHERE (ssdsdba.DataProducer.ParentID_FK = 31548) {noformat} where you use the DataProducerID for the new mooring deployment in the WHERE clause. ==> Change all of the 'File's in the dataContainerType' field to 'Stream's. Mike McCann (First edit: 30 October 2007, Last updated: 1 May 2009)]]> 11240443 11273179 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs and the path to the xml files to the newly deployed deviceIDs (aka ISI_IDs) and new xml directory. Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml wile with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /mbari/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed. h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSDS ingest will recognize them as new. # +Do not close (set endDates) the existing Test deployment until after the oasisToSSDS has run with the new platform deployment name+ and all new instrument deployments have been created in SSDS_Metadata. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadata, e.g.: {noformat} SELECT id, name, startDate, endDate, ParentID_FK FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') ORDER BY id DESC {noformat} Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file. # You will also need to edit the DStoNetCDF.pl script to set the new name of the M1 deployment, e.g.: {noformat} # # SSDS mooring name Deployment lookup # my %ssdsMooringDeplNames = ( M0 => 'M0', M1 => 'M1 - October 2009', M1Test => 'Test M1 - October 2009', M2 => 'M2 - April 2009', {noformat} # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. # When the mooring log message is sent to oasis with the actual time of deployment enter that as the startDate in the database.  All times (DTGs) in the SSDS_Metadata database are GMT. Make sure to set the startDate for the imctd microcat sensors too. This can be done with a query like this where you use the proper dataProduceIDs for the mooring and the imcd: {noformat} SELECT id, name, ParentID_FK, startDate, endDate, nominalDepth FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') AND (ParentID_FK = 31564) OR (ParentID_FK = 31548) ORDER BY id DESC {noformat} # Because of some bug in SSDS ingest the dataContainerTypes of the outputs from the instrument deployments do not get properly assigned the values of 'Stream'.  This needs to be fixed so that the NDBC datatransfers will work. To fix it edit the SSDS_Metadata database starting with a query like this: {noformat} SELECT ssdsdba.DataContainer.* FROM ssdsdba.DataProducer INNER JOIN ssdsdba.DataContainer ON ssdsdba.DataProducer.id = ssdsdba.DataContainer.DataProducerID_FK WHERE (ssdsdba.DataProducer.ParentID_FK = 31548) {noformat} where you use the DataProducerID for the new mooring deployment in the WHERE clause. ==> Change all of the 'File's in the dataContainerType' field to 'Stream's. Mike McCann (First edit: 30 October 2007, Last updated: 1 May 2009)]]> 11240442 16810037 16777269 16810036 16777268 16810049 16777281 16810047 16777279 16810045 16777277 16810051 16777283 9830848 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs to the newly deployed deviceIDs (aka ISI_IDs). Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml wile with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /hosts/tornado_vol0/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed. h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSD ingest will recognize them as new. # +Do not close (set endDates) the existing Test deployment until after the oasisToSSDS has run with the new platform deployment name+ and all new instrument deployments have been created in SSDS_Metadata. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadta, e.g.: {noformat} SELECT id, name, startDate, endDate, ParentID_FK FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') ORDER BY id DESC {noformat} Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file. # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. Mike McCann (First edit: 30 October 2007, Last updated: 29 April 2009)]]> 9798092 1867910 1835146 1867909 1835145 1867908 1835144 1867907 1835143 1867914 1835150 1867913 1835149 1867912 1835148 1867911 1835147 1867917 1835153 1867916 1835152 1867915 1835151 1867887 1835123 1867888 1835124 1867889 1835125 1867890 1835126 1867883 1835119 18645323 18579809 1867885 1835121 1867886 1835122 18645320 18579806 18645319 18579805 18645322 18579808 18645321 18579807 18645316 18579802 18645315 18579801 1867877 1835113 18645318 18579804 1867878 1835114 1867903 1835139 6226523 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. # Edit the mooring .cfg file and changed the instrument deviceIDs to the newly deployed deviceIDs (aka ISI_IDs). Refer to email from OSG for the correct numbers, e.g.: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application, e.g.: {noformat} {noformat} Remove any time or location information and check back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen. Copy the files into the _yyyy_/xml subdirectory. Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file; if you include other attributes they will overwrite what it in the database. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml wile with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /hosts/tornado_vol0/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed. h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSD ingest will recognize them as new. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /hosts/tornado_vol0/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query like this where the DataProducer.id is >= the id of the new platform, e.g.: {noformat} {noformat} # ;sd;s #   #   # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id > 27060) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. #   # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DStoNetCDF.pl \-mooring parameter from 'M?Test' to 'M?', where ? is the mooring number. If the month has changed since the test deployment was started then also update the DDIR shell variable with the correct YYYMM. # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. Mike McCann (First edit: 30 October 2007, Last updated: 8 October 2008)]]> 6259320 1867906 1835142 1867905 1835141 1867900 1835136 1867899 1835135 1867902 1835138 1867901 1835137 1867896 1835132 1867895 1835131 1867898 1835134 1867897 1835133 19071252 19038486 1867892 1835128 1867891 1835127 6226519 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. # Edit the mooring .cfg file and changed the instrument deviceIDs to the newly deployed deviceIDs (aka ISI_IDs). Refer to email from OSG for the correct numbers, e.g.: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application, e.g.: {noformat} {noformat} Remove any time or location information and check back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen. Copy the files into the _yyyy_/xml subdirectory. Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file; if you include other attributes they will overwrite what it in the database. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml wile with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /hosts/tornado_vol0/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. The steps to do this are outlined below. \[As of November 2007, there are several issues with making this a fool-proof set of instructions; therefore, for now they will serve as documentation for tasks  in the 2008 SSDS Hardening project.\] \\ # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id > 27060) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} # Before initiating the oasisToSSDS program you may want to take a printout of the ssds.cfg file over to the actual Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the instrument swap procedure and doing a deep delete on the wrong instrument deployment. # If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DStoNetCDF.pl \-mooring parameter from 'M?Test' to 'M?', where ? is the mooring number. If the month has changed since the test deployment was started then also update the DDIR shell variable with the correct YYYMM. # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. Mike McCann (First edit: 30 October 2007, Last updated: 30 September 2008)]]> 6259316 1867894 1835130 1867893 1835129 6226521 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. # Edit the mooring .cfg file and changed the instrument deviceIDs to the newly deployed deviceIDs (aka ISI_IDs). Refer to email from OSG for the correct numbers, e.g.: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application, e.g.: {noformat} {noformat} Remove any time or location information and check back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen. Copy the files into the _yyyy_/xml subdirectory. Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file; if you include other attributes they will overwrite what it in the database. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml wile with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /hosts/tornado_vol0/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed. {quote}   {quote} The steps to do this are outlined below. \[As of November 2007, there are several issues with making this a fool-proof set of instructions; therefore, for now they will serve as documentation for tasks  in the 2008 SSDS Hardening project.\] \\ # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id > 27060) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} # Before initiating the oasisToSSDS program you may want to take a printout of the ssds.cfg file over to the actual Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the instrument swap procedure and doing a deep delete on the wrong instrument deployment. # If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DStoNetCDF.pl \-mooring parameter from 'M?Test' to 'M?', where ? is the mooring number. If the month has changed since the test deployment was started then also update the DDIR shell variable with the correct YYYMM. # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. Mike McCann (First edit: 30 October 2007, Last updated: 30 September 2008)]]> 6259318 3670018 3637250 3670017 3637249 26673290 26640533 1867853 1835089 26673296 26640539 26673297 26640540 26673292 26640535 26673294 26640537 1867868 1835104 1867867 1835103 1867866 1835102 1867865 1835101 1867864 1835100 9830737 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs to the newly deployed deviceIDs (aka ISI_IDs). Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml wile with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /hosts/tornado_vol0/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed. h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSD ingest will recognize them as new. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadta, e.g.: {noformat} SELECT id, name, startDate, endDate, ParentID_FK FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') ORDER BY id DESC {noformat} Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file. # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. Mike McCann (First edit: 30 October 2007, Last updated: 29 April 2009)]]> 9797971 1867863 1835099 1867862 1835098 1867860 1835096 1835105 1867869 1 2007-10-03 12:37:04.963 2007-10-03 12:37:04.963 1835089 9830713 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs to the newly deployed deviceIDs (aka ISI_IDs). Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml wile with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /hosts/tornado_vol0/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed. h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSD ingest will recognize them as new. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /hosts/tornado_vol0/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query like this where the DataProducer.id is >= the id of the new platform, e.g.: {noformat} {noformat} # ;sd;s #   #   # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. #   # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DPLOYMENTS file. # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. Mike McCann (First edit: 30 October 2007, Last updated: 8 October 2008)]]> 9797946 17891549 17858805 11273004 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs to the newly deployed deviceIDs (aka ISI_IDs). Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml wile with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /mbari/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed. h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSDS ingest will recognize them as new. # +Do not close (set endDates) the existing Test deployment until after the oasisToSSDS has run with the new platform deployment name+ and all new instrument deployments have been created in SSDS_Metadata. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadata, e.g.: {noformat} SELECT id, name, startDate, endDate, ParentID_FK FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') ORDER BY id DESC {noformat} Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file. # You will also need to edit the DStoNetCDF.pl script to set the new name of the M1 deployment, e.g.: {noformat} # # SSDS mooring name Deployment lookup # my %ssdsMooringDeplNames = ( M0 => 'M0', M1 => 'M1 - October 2009', M1Test => 'Test M1 - October 2009', M2 => 'M2 - April 2009', {noformat} # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. # When the mooring log message is sent to oasis with the actual time of deployment enter that as the startDate in the database.  All times (DTGs) in the SSDS_Metadata database are GMT. Make sure to set the startDate for the imctd microcat sensors too. This can be done with a query like this where you use the proper dataProduceIDs for the mooring and the imcd: {noformat} SELECT id, name, ParentID_FK, startDate, endDate, nominalDepth FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') AND (ParentID_FK = 31564) OR (ParentID_FK = 31548) ORDER BY id DESC {noformat} # Because of some bug in SSDS ingest the dataContainerTypes of the outputs from the instrument deployments do not get properly assigned the values of 'Stream'.  This needs to be fixed so that the NDBC datatransfers will work. To fix it edit the SSDS_Metadata database starting with a query like this: {noformat} SELECT ssdsdba.DataContainer.* FROM ssdsdba.DataProducer INNER JOIN ssdsdba.DataContainer ON ssdsdba.DataProducer.id = ssdsdba.DataContainer.DataProducerID_FK WHERE (ssdsdba.DataProducer.ParentID_FK = 31548) {noformat} where you use the DataProducerID for the new mooring deployment in the WHERE clause. ==> Change all of the 'File's in the dataContainerType' field to 'Stream's. Mike McCann (First edit: 30 October 2007, Last updated: 1 May 2009)]]> 11240264 17891554 17858810 11273006 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs to the newly deployed deviceIDs (aka ISI_IDs). Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml wile with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /mbari/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed. h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSDS ingest will recognize them as new. # +Do not close (set endDates) the existing Test deployment until after the oasisToSSDS has run with the new platform deployment name+ and all new instrument deployments have been created in SSDS_Metadata. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadata, e.g.: {noformat} SELECT id, name, startDate, endDate, ParentID_FK FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') ORDER BY id DESC {noformat} Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file. # You will also need to edit the DStoNetCDF.pl script to set the new name of the M1 deployment, e.g.: {noformat} # # SSDS mooring name Deployment lookup # my %ssdsMooringDeplNames = ( M0 => 'M0', M1 => 'M1 - October 2009', M1Test => 'Test M1 - October 2009', M2 => 'M2 - April 2009', {noformat} # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. # When the mooring log message is sent to oasis with the actual time of deployment enter that as the startDate in the database.  All times (DTGs) in the SSDS_Metadata database are GMT. Make sure to set the startDate for the imctd microcat sensors too. This can be done with a query like this where you use the proper dataProduceIDs for the mooring and the imcd: {noformat} SELECT id, name, ParentID_FK, startDate, endDate, nominalDepth FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') AND (ParentID_FK = 31564) OR (ParentID_FK = 31548) ORDER BY id DESC {noformat} # Because of some bug in SSDS ingest the dataContainerTypes of the outputs from the instrument deployments do not get properly assigned the values of 'Stream'.  This needs to be fixed so that the NDBC datatransfers will work. To fix it edit the SSDS_Metadata database starting with a query like this: {noformat} SELECT ssdsdba.DataContainer.* FROM ssdsdba.DataProducer INNER JOIN ssdsdba.DataContainer ON ssdsdba.DataProducer.id = ssdsdba.DataContainer.DataProducerID_FK WHERE (ssdsdba.DataProducer.ParentID_FK = 31548) {noformat} where you use the DataProducerID for the new mooring deployment in the WHERE clause. ==> Change all of the 'File's in the dataContainerType' field to 'Stream's. Mike McCann (First edit: 30 October 2007, Last updated: 1 May 2009)]]> 11240266 6226464 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. # Edit the mooring .cfg file and changed the instrument deviceIDs to the newly deployed deviceIDs (aka ISI_IDs). Refer to email from OSG for the correct numbers, e.g.: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application, e.g.: {noformat} {noformat} Remove any time or location information and check back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen. Copy the files into the _yyyy_/xml subdirectory. Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file; if you include other attributes they will overwrite what it in the database. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml wile with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /hosts/tornado_vol0/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. The steps to do this are outlined below. \[As of November 2007, there are several issues with making this a fool-proof set of instructions; therefore, for now they will serve as documentation for tasks  in the 2008 SSDS Hardening project.\] \\ # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id > 27060) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} # Before initiating the oasisToSSDS program you may want to take a printout of the ssds.cfg file over to the actual Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the instrument swap procedure and doing a deep delete on the wrong instrument deployment. # If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DStoNetCDF.pl \-mooring parameter from 'M?Test' to 'M?', where ? is the mooring number. If the month has changed since the test deployment was started then also update the DDIR shell variable with the correct YYYMM. # Monitor the SSDS processing web page (e.g. http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html) for successful data processing. Mike McCann (First edit: 30 October 2007, Last updated: 30 September 2008)]]> 6259254 6226466 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. # Edit the mooring .cfg file and changed the instrument deviceIDs to the newly deployed deviceIDs (aka ISI_IDs). Refer to email from OSG for the correct numbers, e.g.: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application, e.g.: {noformat} {noformat} Remove any time or location information and check back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen. Copy the files into the _yyyy_/xml subdirectory. Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file; if you include other attributes they will overwrite what it in the database. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml wile with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /hosts/tornado_vol0/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. The steps to do this are outlined below. \[As of November 2007, there are several issues with making this a fool-proof set of instructions; therefore, for now they will serve as documentation for tasks  in the 2008 SSDS Hardening project.\] \\ # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id > 27060) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} # Before initiating the oasisToSSDS program you may want to take a printout of the ssds.cfg file over to the actual Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the instrument swap procedure and doing a deep delete on the wrong instrument deployment. # If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DStoNetCDF.pl \-mooring parameter from 'M?Test' to 'M?', where ? is the mooring number. If the month has changed since the test deployment was started then also update the DDIR shell variable with the correct YYYMM. # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. Mike McCann (First edit: 30 October 2007, Last updated: 30 September 2008)]]> 6259256 11273012 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs to the newly deployed deviceIDs (aka ISI_IDs). Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml wile with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /mbari/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed. h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSDS ingest will recognize them as new. # +Do not close (set endDates) the existing Test deployment until after the oasisToSSDS has run with the new platform deployment name+ and all new instrument deployments have been created in SSDS_Metadata. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadata, e.g.: {noformat} SELECT id, name, startDate, endDate, ParentID_FK FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') ORDER BY id DESC {noformat} Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file. # You will also need to edit the DStoNetCDF.pl script to set the new name of the M1 deployment, e.g.: {noformat} # # SSDS mooring name Deployment lookup # my %ssdsMooringDeplNames = ( M0 => 'M0', M1 => 'M1 - October 2009', M1Test => 'Test M1 - October 2009', M2 => 'M2 - April 2009', {noformat} # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. # When the mooring log message is sent to oasis with the actual time of deployment enter that as the startDate in the database.  All times (DTGs) in the SSDS_Metadata database are GMT. Make sure to set the startDate for the imctd microcat sensors too. This can be done with a query like this where you use the proper dataProduceIDs for the mooring and the imcd: {noformat} SELECT id, name, ParentID_FK, startDate, endDate, nominalDepth FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') AND (ParentID_FK = 31564) OR (ParentID_FK = 31548) ORDER BY id DESC {noformat} # Because of some bug in SSDS ingest the dataContainerTypes of the outputs from the instrument deployments do not get properly assigned the values of 'Stream'.  This needs to be fixed so that the NDBC datatransfers will work. To fix it edit the SSDS_Metadata database starting with a query like this: {noformat} SELECT ssdsdba.DataContainer.* FROM ssdsdba.DataProducer INNER JOIN ssdsdba.DataContainer ON ssdsdba.DataProducer.id = ssdsdba.DataContainer.DataProducerID_FK WHERE (ssdsdba.DataProducer.ParentID_FK = 31548) {noformat} where you use the DataProducerID for the new mooring deployment in the WHERE clause. ==> Change all of the 'File's in the dataContainerType' field to 'Stream's. Mike McCann (First edit: 30 October 2007, Last updated: 1 May 2009)]]> 11240272 6226462 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. # Edit the mooring .cfg file and changed the instrument deviceIDs to the newly deployed deviceIDs (aka ISI_IDs). Refer to email from OSG for the correct numbers, e.g.: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application, e.g.: {noformat} {noformat} Remove any time or location information and check back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen. Copy the files into the _yyyy_/xml subdirectory. Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file; if you include other attributes they will overwrite what it in the database. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml wile with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /hosts/tornado_vol0/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ##  Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. The steps to do this are outlined below. \[As of November 2007, there are several issues with making this a fool-proof set of instructions; therefore, for now they will serve as documentation for tasks  in the 2008 SSDS Hardening project.\] \\ # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id > 27060) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the instrument swap procedure and doing a deep delete on the wrong instrument deployment. If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: <251 elvis.shore.mbari.org /u/ssdsadmin> dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02 \- or wait for it to run with the hourly download on tsunami. #   #   #   Mike McCann (30 October 2007)]]> 6259252 17268840 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs and the path to the xml files to the newly deployed deviceIDs (aka ISI_IDs) and new xml directory. Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen or JEdit. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml file with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. \[Note: With the M1 - October 2009 deployment we added the individual inductive modem microcats as children of the mooring and also configured a parallel deployment of the TString that produced the same data. With the M2 - April 2010 deployment we configured the individual IM mirocrocats and did not configure a parallel TString deployment. This is a better model for the system and is much easier to configure and produces more easily consumed data by downstream processes such as combineTS.pl.\] With future deployments we will not configure TString. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /mbari/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. \\ For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed (or...as below \-rschramm 4/2010) \\ \\ {noformat} declare @myID as bigint set @myID = 31548 SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE id = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk in ( SELECT id FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL) ORDER BY id {noformat} h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSDS ingest will recognize them as new. # +Do not close (set endDates) the existing Test deployment until after the oasisToSSDS has run with the new platform deployment name+ and all new instrument deployments have been created in SSDS_Metadata. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadata, e.g.: {noformat} SELECT id, name, startDate, endDate, ParentID_FK FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') ORDER BY id DESC {noformat} Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file. # You will also need to edit the DStoNetCDF.pl script to set the new name of the M1 deployment, e.g.: {noformat} # # SSDS mooring name Deployment lookup # my %ssdsMooringDeplNames = ( M0 => 'M0', M1 => 'M1 - October 2009', M1Test => 'Test M1 - October 2009', M2 => 'M2 - April 2009', {noformat} # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. # When the mooring log message is sent to oasis with the actual time of deployment enter that as the startDate in the database.  All times (DTGs) in the SSDS_Metadata database are GMT. Make sure to set the startDate for the imctd microcat sensors too. This can be done with a query like this where you use the proper dataProduceIDs for the mooring and the imcd: {noformat} SELECT id, name, ParentID_FK, startDate, endDate, nominalDepth FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') AND (ParentID_FK = 31564) OR (ParentID_FK = 31548) ORDER BY id DESC {noformat} # Because of some bug in SSDS ingest the dataContainerTypes of the outputs from the instrument deployments do not get properly assigned the values of 'Stream'.  This needs to be fixed so that the NDBC datatransfers will work. To fix it edit the SSDS_Metadata database starting with a query like this: {noformat} SELECT ssdsdba.DataContainer.* FROM ssdsdba.DataProducer INNER JOIN ssdsdba.DataContainer ON ssdsdba.DataProducer.id = ssdsdba.DataContainer.DataProducerID_FK WHERE (ssdsdba.DataProducer.ParentID_FK = 31548) {noformat} where you use the DataProducerID for the new mooring deployment in the WHERE clause. ==> Change all of the 'File's in the dataContainerType' field to 'Stream's. h2. C. Procedures to be done after the mooring turn h4. Set up download info deployment and reporting Though not a real instrument, we configure a virtual 'dlinfo' instrument for the download scripts to attach download statistics data. We re-use the same device IDs for the M1 and M2 moorings, so it's best to configure this after the new mooring is out and all of those deployments have been closed. As the data are delivered "out of band" from the typical OASIS instruments we need to create a deployment for dlinfo instrument by hand. # The most direct way is to use the createDuplicateDeepDeployment service call. For example, to duplicate the 2009 M2 dlinfo deployment for the 2010 M2 deployment this call was executed: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer\|id=*34496*&p2Type=Date&p2Value=*2010-04-03T22:00:00Z*&p3Type=boolean&p3Value=false&p4Type=Date&p4Value=*2010-04-03T23:00:00Z*&p5Type=String&p5Value=getM2-download&p6Type=String&p6Value=&delimiter=\|. Of course you will need to adjust the DataProducer ID Date values for the new deployment you are creating. Here is the Key to the parameters: {noformat} Key: ---- http://localhost:8080/servlet/MetadataAccessServlet ?responseType=text &delimiter=| &objectToInvokeOn=DataProducerAccess &method=createDuplicateDeepDeployment &p1Type=DataProducer &p1Value=DataProducer|id=XXXX (XXXX is the ID of the deployment to copy) &p2Type=Date &p2Value=XXXXXX (XXXXXX is the start date of the new copy in XML format YYYY-MM-DDTHH:MM:SSZ) &p3Type=boolean &p3Value=(true|false) (this is to indicate if you want the original deployment to be closed) &p4Type=Date &p4Value=XXXXXX (XXXXXX is the end date for the original deployment (if p3Value is true)) &p5Type=String &p5Value=XXXXXX (XXXXXX is the DataProducer name of the new DataProducer) &p6Type=String &p6Value=XXXXXX (XXXXXX is the base URL to use for the new DataContainers that will be created). &delimiter=| {noformat}\\ Executing the createDuplicateDeepDeployment service call will return an ID for the new deployment. # Edit the new deployment record in the DataProducer table to adjust it's parentID_FK to be for the new mooring. While there edit times and name as appropriate. # Edit the getM? script on tsunami to use the proper device and parent ID (this will be the device ID of the torroid of the mooring deployment) and make sure that the '/oasis/bin/ssdsSubmit.pl $deviceId $parentId "$starttime_es,$endtime_es,$filesize,$rtnsts"' line in the script is configured to run. Here's another example for the October 2010 M1 turn: {noformat} http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&\ p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer|id=35591&p2Type=Date&p2Value=2010-10-27T21:00:00Z&p3Type=boolean&p3Value=false&p4Type=Date&\ p4Value=2010-q0-27T20:30:00Z&p5Type=String&p5Value=getM1-download&p6Type=String&p6Value=&delimiter=| (This returned a new DataProducerID of 41556. The ParentID_FK for this record in the DataProducer table was then changed from 35388 to 41197 and the endDate cleared to null. Also, with the change to alternating deviceIDs implemented in 2010 the deviceID_FK was changed from 1698 to 1768 for this M! deployment. {noformat} N.B. A rotating scheme for the virtual device IDs was implemented in 2010. This permits the steps described above to be executed prior to the mooring turn during the extensive dock-side test period. h4. Enable processing for other "virtual" devices # Turn on ClockSync processing. We also use a virtual device ID for these data. Simply uncomment the line for it in the ssds.cfg file. # Turn on ISUS processing. Simply uncomment the line for it in the ssds.cfg file. (Device ID re-used from previous deployment.) h4. Cycle links to previous deployment # Edit previous.html file in /mbari/ssdsdata/deployments to add a line for the new deployment and add the end date and archive url for the just closed deployment, e.g.: {noformat}
  • 200904 2009-04-29 to 2010-04-03
  • 201004 2010-04-03 to present
    • {noformat}\\ # Add lines to DEPLOYMENTS file in dev/DPforSSDS/cimt/ for the just closed deployment, e.g.: {noformat} DStoNetCDF.pl -mooring M2 -deployment "M2 - April 2009" -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org -outputDir /mbari/ssdsdata/deployments -procClosed -verbose combineM.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments combineTS.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments combineAll.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments and /bin/cp /mbari/ssdsdata/deployments/m2/200904/OS_*.nc /mbari/FTP/pub/OceanSITES {noformat}\\ # Execute these lines to generate "closed deployment" data products and web pages and for submission to the OceanSITES GDAC Mike McCann (First edit: 30 October 2007, Last updated: 1 May 2009)]]>     17236075     17268842 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs and the path to the xml files to the newly deployed deviceIDs (aka ISI_IDs) and new xml directory. Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen or JEdit. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml file with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. \[Note: With the M1 - October 2009 deployment we added the individual inductive modem microcats as children of the mooring and also configured a parallel deployment of the TString that produced the same data. With the M2 - April 2010 deployment we configured the individual IM mirocrocats and did not configure a parallel TString deployment. This is a better model for the system and is much easier to configure and produces more easily consumed data by downstream processes such as combineTS.pl.\] With future deployments we will not configure TString. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /mbari/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. \\ For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed (or...as below \-rschramm 4/2010) \\ \\ {noformat} declare @myID as bigint set @myID = 31548 SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE id = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk in ( SELECT id FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL) ORDER BY id {noformat} h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSDS ingest will recognize them as new. # +Do not close (set endDates) the existing Test deployment until after the oasisToSSDS has run with the new platform deployment name+ and all new instrument deployments have been created in SSDS_Metadata. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadata, e.g.: {noformat} SELECT id, name, startDate, endDate, ParentID_FK FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') ORDER BY id DESC {noformat} Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file. # You will also need to edit the DStoNetCDF.pl script to set the new name of the M1 deployment, e.g.: {noformat} # # SSDS mooring name Deployment lookup # my %ssdsMooringDeplNames = ( M0 => 'M0', M1 => 'M1 - October 2009', M1Test => 'Test M1 - October 2009', M2 => 'M2 - April 2009', {noformat} # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. # When the mooring log message is sent to oasis with the actual time of deployment enter that as the startDate in the database.  All times (DTGs) in the SSDS_Metadata database are GMT. Make sure to set the startDate for the imctd microcat sensors too. This can be done with a query like this where you use the proper dataProduceIDs for the mooring and the imcd: {noformat} SELECT id, name, ParentID_FK, startDate, endDate, nominalDepth FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') AND (ParentID_FK = 31564) OR (ParentID_FK = 31548) ORDER BY id DESC {noformat} # Because of some bug in SSDS ingest the dataContainerTypes of the outputs from the instrument deployments do not get properly assigned the values of 'Stream'.  This needs to be fixed so that the NDBC datatransfers will work. To fix it edit the SSDS_Metadata database starting with a query like this: {noformat} SELECT ssdsdba.DataContainer.* FROM ssdsdba.DataProducer INNER JOIN ssdsdba.DataContainer ON ssdsdba.DataProducer.id = ssdsdba.DataContainer.DataProducerID_FK WHERE (ssdsdba.DataProducer.ParentID_FK = 31548) {noformat} where you use the DataProducerID for the new mooring deployment in the WHERE clause. ==> Change all of the 'File's in the dataContainerType' field to 'Stream's. h2. C. Procedures to be done after the mooring turn h4. Set up download info deployment and reporting Though not a real instrument, we configure a virtual 'dlinfo' instrument for the download scripts to attach download statistics data. We re-use the same device IDs for the M1 and M2 moorings, so it's best to configure this after the new mooring is out and all of those deployments have been closed. As the data are delivered "out of band" from the typical OASIS instruments we need to create a deployment for dlinfo instrument by hand. # The most direct way is to use the createDuplicateDeepDeployment service call. For example, to duplicate the 2009 M2 dlinfo deployment for the 2010 M2 deployment this call was executed: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer\|id=*34496*&p2Type=Date&p2Value=*2010-04-03T22:00:00Z*&p3Type=boolean&p3Value=false&p4Type=Date&p4Value=*2010-04-03T23:00:00Z*&p5Type=String&p5Value=getM2-download&p6Type=String&p6Value=&delimiter=\|. Of course you will need to adjust the DataProducer ID Date values for the new deployment you are creating. Here is the Key to the parameters: {noformat} Key: ---- http://localhost:8080/servlet/MetadataAccessServlet ?responseType=text &delimiter=| &objectToInvokeOn=DataProducerAccess &method=createDuplicateDeepDeployment &p1Type=DataProducer &p1Value=DataProducer|id=XXXX (XXXX is the ID of the deployment to copy) &p2Type=Date &p2Value=XXXXXX (XXXXXX is the start date of the new copy in XML format YYYY-MM-DDTHH:MM:SSZ) &p3Type=boolean &p3Value=(true|false) (this is to indicate if you want the original deployment to be closed) &p4Type=Date &p4Value=XXXXXX (XXXXXX is the end date for the original deployment (if p3Value is true)) &p5Type=String &p5Value=XXXXXX (XXXXXX is the DataProducer name of the new DataProducer) &p6Type=String &p6Value=XXXXXX (XXXXXX is the base URL to use for the new DataContainers that will be created). &delimiter=| {noformat}\\ Executing the createDuplicateDeepDeployment service call will return an ID for the new deployment. # Edit the new deployment record in the DataProducer table to adjust it's parentID_FK to be for the new mooring. While there edit times and name as appropriate. # Edit the getM? script on tsunami to use the proper device and parent ID (this will be the device ID of the torroid of the mooring deployment) and make sure that the '/oasis/bin/ssdsSubmit.pl $deviceId $parentId "$starttime_es,$endtime_es,$filesize,$rtnsts"' line in the script is configured to run. Here's another example for the October 2010 M1 turn: {noformat} http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&\ p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer|id=35591&p2Type=Date&p2Value=2010-10-27T21:00:00Z&p3Type=boolean&p3Value=false&p4Type=Date&\ p4Value=2010-q0-27T20:30:00Z&p5Type=String&p5Value=getM1-download&p6Type=String&p6Value=&delimiter=| (This returned a new DataProducerID of 41556. The ParentID_FK for this record in the DataProducer table was then changed from 35388 to 41197 and the endDate cleared to null. Also, with the change to alternating deviceIDs implemented in 2010 the deviceID_FK was changed from 1698 to 1768 for this M! deployment. {noformat} N.B. A rotating scheme for the virtual device IDs was implemented in 2010. This permits the steps described above to be executed prior to the mooring turn during the extensive dock-side test period. h4. Enable processing for other "virtual" devices # Turn on ClockSync processing. We also use a virtual device ID for these data. Simply uncomment the line for it in the ssds.cfg file. # Turn on ISUS processing. Simply uncomment the line for it in the ssds.cfg file. (Device ID re-used from previous deployment.) h4. Make all of the child instrument deployment start times the same as the mooring start time When SSDS receives a packet from an instrument that is not currently deployed it will create a Deployment record (in the DataProducer table) with a startDate that is set to the time of the first record received. As the mooring starts up all of the instrument deployments will have different start times based on when they each first saw data. This can present problems for the data processing that follows, especially for the jobs that aggregate the microcat data into a single ZT file that is used to produce the contour temperature and salinity wind stick plots. To prevent these problems it's best to edit the startDates of the child instrument deployments so that they are all the same. This is currently most easily done through Enterprise Manager with a query like below (this is for the 201010 M1 deployment) to get all the child instrument deployments: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 41197) ORDER BY startDate {noformat} Then copy and paste the datetime string from one cell to the next. The times are GMT. h4. Cycle links to previous deployment # Edit previous.html file in /mbari/ssdsdata/deployments to add a line for the new deployment and add the end date and archive url for the just closed deployment, e.g.: {noformat}
  • 200904 2009-04-29 to 2010-04-03
  • 201004 2010-04-03 to present
    • {noformat}\\ # Add lines to DEPLOYMENTS file in dev/DPforSSDS/cimt/ for the just closed deployment, e.g.: {noformat} DStoNetCDF.pl -mooring M2 -deployment "M2 - April 2009" -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org -outputDir \ /mbari/ssdsdata/deployments -procClosed -verbose combineM.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments combineTS.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments combineAll.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments and /bin/cp /mbari/ssdsdata/deployments/m2/200904/OS_*.nc /mbari/FTP/pub/OceanSITES {noformat}\\ # Execute these lines to generate "closed deployment" data products and web pages and for submission to the OceanSITES GDAC Mike McCann (First edit: 30 October 2007, Last updated: 1 May 2009)]]>     17236077     17268841 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs and the path to the xml files to the newly deployed deviceIDs (aka ISI_IDs) and new xml directory. Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen or JEdit. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml file with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. \[Note: With the M1 - October 2009 deployment we added the individual inductive modem microcats as children of the mooring and also configured a parallel deployment of the TString that produced the same data. With the M2 - April 2010 deployment we configured the individual IM mirocrocats and did not configure a parallel TString deployment. This is a better model for the system and is much easier to configure and produces more easily consumed data by downstream processes such as combineTS.pl.\] With future deployments we will not configure TString. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /mbari/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. \\ For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed (or...as below \-rschramm 4/2010) \\ \\ {noformat} declare @myID as bigint set @myID = 31548 SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE id = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk in ( SELECT id FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL) ORDER BY id {noformat} h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSDS ingest will recognize them as new. # +Do not close (set endDates) the existing Test deployment until after the oasisToSSDS has run with the new platform deployment name+ and all new instrument deployments have been created in SSDS_Metadata. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadata, e.g.: {noformat} SELECT id, name, startDate, endDate, ParentID_FK FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') ORDER BY id DESC {noformat} Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file. # You will also need to edit the DStoNetCDF.pl script to set the new name of the M1 deployment, e.g.: {noformat} # # SSDS mooring name Deployment lookup # my %ssdsMooringDeplNames = ( M0 => 'M0', M1 => 'M1 - October 2009', M1Test => 'Test M1 - October 2009', M2 => 'M2 - April 2009', {noformat} # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. # When the mooring log message is sent to oasis with the actual time of deployment enter that as the startDate in the database.  All times (DTGs) in the SSDS_Metadata database are GMT. Make sure to set the startDate for the imctd microcat sensors too. This can be done with a query like this where you use the proper dataProduceIDs for the mooring and the imcd: {noformat} SELECT id, name, ParentID_FK, startDate, endDate, nominalDepth FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') AND (ParentID_FK = 31564) OR (ParentID_FK = 31548) ORDER BY id DESC {noformat} # Because of some bug in SSDS ingest the dataContainerTypes of the outputs from the instrument deployments do not get properly assigned the values of 'Stream'.  This needs to be fixed so that the NDBC datatransfers will work. To fix it edit the SSDS_Metadata database starting with a query like this: {noformat} SELECT ssdsdba.DataContainer.* FROM ssdsdba.DataProducer INNER JOIN ssdsdba.DataContainer ON ssdsdba.DataProducer.id = ssdsdba.DataContainer.DataProducerID_FK WHERE (ssdsdba.DataProducer.ParentID_FK = 31548) {noformat} where you use the DataProducerID for the new mooring deployment in the WHERE clause. ==> Change all of the 'File's in the dataContainerType' field to 'Stream's. h2. C. Procedures to be done after the mooring turn h4. Set up download info deployment and reporting Though not a real instrument, we configure a virtual 'dlinfo' instrument for the download scripts to attach download statistics data. We re-use the same device IDs for the M1 and M2 moorings, so it's best to configure this after the new mooring is out and all of those deployments have been closed. As the data are delivered "out of band" from the typical OASIS instruments we need to create a deployment for dlinfo instrument by hand. # The most direct way is to use the createDuplicateDeepDeployment service call. For example, to duplicate the 2009 M2 dlinfo deployment for the 2010 M2 deployment this call was executed: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer\|id=*34496*&p2Type=Date&p2Value=*2010-04-03T22:00:00Z*&p3Type=boolean&p3Value=false&p4Type=Date&p4Value=*2010-04-03T23:00:00Z*&p5Type=String&p5Value=getM2-download&p6Type=String&p6Value=&delimiter=\|. Of course you will need to adjust the DataProducer ID Date values for the new deployment you are creating. Here is the Key to the parameters: {noformat} Key: ---- http://localhost:8080/servlet/MetadataAccessServlet ?responseType=text &delimiter=| &objectToInvokeOn=DataProducerAccess &method=createDuplicateDeepDeployment &p1Type=DataProducer &p1Value=DataProducer|id=XXXX (XXXX is the ID of the deployment to copy) &p2Type=Date &p2Value=XXXXXX (XXXXXX is the start date of the new copy in XML format YYYY-MM-DDTHH:MM:SSZ) &p3Type=boolean &p3Value=(true|false) (this is to indicate if you want the original deployment to be closed) &p4Type=Date &p4Value=XXXXXX (XXXXXX is the end date for the original deployment (if p3Value is true)) &p5Type=String &p5Value=XXXXXX (XXXXXX is the DataProducer name of the new DataProducer) &p6Type=String &p6Value=XXXXXX (XXXXXX is the base URL to use for the new DataContainers that will be created). &delimiter=| {noformat}\\ Executing the createDuplicateDeepDeployment service call will return an ID for the new deployment. # Edit the new deployment record in the DataProducer table to adjust it's parentID_FK to be for the new mooring. While there edit times and name as appropriate. # Edit the getM? script on tsunami to use the proper device and parent ID (this will be the device ID of the torroid of the mooring deployment) and make sure that the '/oasis/bin/ssdsSubmit.pl $deviceId $parentId "$starttime_es,$endtime_es,$filesize,$rtnsts"' line in the script is configured to run. Here's another example for the October 2010 M1 turn: {noformat} http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&\ p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer|id=35591&p2Type=Date&p2Value=2010-10-27T21:00:00Z&p3Type=boolean&p3Value=false&p4Type=Date&\ p4Value=2010-q0-27T20:30:00Z&p5Type=String&p5Value=getM1-download&p6Type=String&p6Value=&delimiter=| (This returned a new DataProducerID of 41556. The ParentID_FK for this record in the DataProducer table was then changed from 35388 to 41197 and the endDate cleared to null. Also, with the change to alternating deviceIDs implemented in 2010 the deviceID_FK was changed from 1698 to 1768 for this M! deployment. {noformat} N.B. A rotating scheme for the virtual device IDs was implemented in 2010. This permits the steps described above to be executed prior to the mooring turn during the extensive dock-side test period. h4. Enable processing for other "virtual" devices # Turn on ClockSync processing. We also use a virtual device ID for these data. Simply uncomment the line for it in the ssds.cfg file. # Turn on ISUS processing. Simply uncomment the line for it in the ssds.cfg file. (Device ID re-used from previous deployment.) h4. Make all of the child instrument deployment start times the same as the mooring start time When SSDS receives a packet from an instrument that is not currently deployed it will create a Deployment record (in the DataProducer table) with a startDate that is set to the time of the first record received. As the mooring starts up all of the instrument deployments will have different start times based on when they each first saw data. This can present problems for the data processing that follows, especially for the jobs that aggregate the microcat data into a single ZT file that is used to produce the contour temperature and salinity wind stick plots. To prevent these problems it's best to edit the startDates of the child instrument deployments so that they are all the same. This is currently most easily done through Enterprise Manager with a query like below (this is for the 201010 M1 deployment) to get all the child instrument deployments: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 41197) ORDER BY startDate {noformat} Then copy and paste the datetime string from one cell to the next. The times are GMT. h4. Cycle links to previous deployment # Edit previous.html file in /mbari/ssdsdata/deployments to add a line for the new deployment and add the end date and archive url for the just closed deployment, e.g.: {noformat}
  • 200904 2009-04-29 to 2010-04-03
  • 201004 2010-04-03 to present
    • {noformat}\\ # Add lines to DEPLOYMENTS file in dev/DPforSSDS/cimt/ for the just closed deployment, e.g.: {noformat} DStoNetCDF.pl -mooring M2 -deployment "M2 - April 2009" -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org -outputDir /mbari/ssdsdata/deployments -procClosed -verbose combineM.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments combineTS.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments combineAll.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments and /bin/cp /mbari/ssdsdata/deployments/m2/200904/OS_*.nc /mbari/FTP/pub/OceanSITES {noformat}\\ # Execute these lines to generate "closed deployment" data products and web pages and for submission to the OceanSITES GDAC Mike McCann (First edit: 30 October 2007, Last updated: 1 May 2009)]]>     17236076     18645294 18579779 18645293 18579778 18645292 18579777 18645298 18579783 18285122 18252364 18645297 18579782 18645295 18579780 18645286 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs and the path to the xml files to the newly deployed deviceIDs (aka ISI_IDs) and new xml directory. Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen or JEdit. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml file with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. \[Note: With the M1 - October 2009 deployment we added the individual inductive modem microcats as children of the mooring and also configured a parallel deployment of the TString that produced the same data. With the M2 - April 2010 deployment we configured the individual IM mirocrocats and did not configure a parallel TString deployment. This is a better model for the system and is much easier to configure and produces more easily consumed data by downstream processes such as combineTS.pl.\] With future deployments we will not configure TString. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /mbari/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. \\ For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed (or...as below \-rschramm 4/2010) \\ \\ {noformat} declare @myID as bigint set @myID = 31548 SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE id = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk in ( SELECT id FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL) ORDER BY id {noformat} h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSDS ingest will recognize them as new. # +Do not close (set endDates) the existing Test deployment until after the oasisToSSDS has run with the new platform deployment name+ and all new instrument deployments have been created in SSDS_Metadata. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadata, e.g.: {noformat} SELECT id, name, startDate, endDate, ParentID_FK FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') ORDER BY id DESC {noformat} Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file. # You will also need to edit the DStoNetCDF.pl script to set the new name of the M1 deployment, e.g.: {noformat} # # SSDS mooring name Deployment lookup # my %ssdsMooringDeplNames = ( M0 => 'M0', M1 => 'M1 - October 2009', M1Test => 'Test M1 - October 2009', M2 => 'M2 - April 2009', {noformat} # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. # When the mooring log message is sent to oasis with the actual time of deployment enter that as the startDate in the database.  All times (DTGs) in the SSDS_Metadata database are GMT. Make sure to set the startDate for the imctd microcat sensors too. This can be done with a query like this where you use the proper dataProduceIDs for the mooring and the imcd: {noformat} SELECT id, name, ParentID_FK, startDate, endDate, nominalDepth FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') AND (ParentID_FK = 31564) OR (ParentID_FK = 31548) ORDER BY id DESC {noformat} # Because of some bug in SSDS ingest the dataContainerTypes of the outputs from the instrument deployments do not get properly assigned the values of 'Stream'.  This needs to be fixed so that the NDBC datatransfers will work. To fix it edit the SSDS_Metadata database starting with a query like this: {noformat} SELECT ssdsdba.DataContainer.* FROM ssdsdba.DataProducer INNER JOIN ssdsdba.DataContainer ON ssdsdba.DataProducer.id = ssdsdba.DataContainer.DataProducerID_FK WHERE (ssdsdba.DataProducer.ParentID_FK = 31548) {noformat} where you use the DataProducerID for the new mooring deployment in the WHERE clause. ==> Change all of the 'File's in the dataContainerType' field to 'Stream's. h2. C. Procedures to be done after the mooring turn h4. Set up download info deployment and reporting Though not a real instrument, we configure a virtual 'dlinfo' instrument for the download scripts to attach download statistics data. We re-use the same device IDs for the M1 and M2 moorings, so it's best to configure this after the new mooring is out and all of those deployments have been closed. As the data are delivered "out of band" from the typical OASIS instruments we need to create a deployment for dlinfo instrument by hand. # The most direct way is to use the createDuplicateDeepDeployment service call. For example, to duplicate the 2009 M2 dlinfo deployment for the 2010 M2 deployment this call was executed: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer\|id=*34496*&p2Type=Date&p2Value=*2010-04-03T22:00:00Z*&p3Type=boolean&p3Value=false&p4Type=Date&p4Value=*2010-04-03T23:00:00Z*&p5Type=String&p5Value=getM2-download&p6Type=String&p6Value=&delimiter=\|. Of course you will need to adjust the DataProducer ID Date values for the new deployment you are creating. Here is the Key to the parameters: {noformat} Key: ---- http://localhost:8080/servlet/MetadataAccessServlet ?responseType=text &delimiter=| &objectToInvokeOn=DataProducerAccess &method=createDuplicateDeepDeployment &p1Type=DataProducer &p1Value=DataProducer|id=XXXX (XXXX is the ID of the deployment to copy) &p2Type=Date &p2Value=XXXXXX (XXXXXX is the start date of the new copy in XML format YYYY-MM-DDTHH:MM:SSZ) &p3Type=boolean &p3Value=(true|false) (this is to indicate if you want the original deployment to be closed) &p4Type=Date &p4Value=XXXXXX (XXXXXX is the end date for the original deployment (if p3Value is true)) &p5Type=String &p5Value=XXXXXX (XXXXXX is the DataProducer name of the new DataProducer) &p6Type=String &p6Value=XXXXXX (XXXXXX is the base URL to use for the new DataContainers that will be created). &delimiter=| {noformat}\\ Executing the createDuplicateDeepDeployment service call will return an ID for the new deployment. # Edit the new deployment record in the DataProducer table to adjust it's parentID_FK to be for the new mooring. While there edit times and name as appropriate. # Edit the getM? script on tsunami to use the proper device and parent ID (this will be the device ID of the torroid of the mooring deployment) and make sure that the '/oasis/bin/ssdsSubmit.pl $deviceId $parentId "$starttime_es,$endtime_es,$filesize,$rtnsts"' line in the script is configured to run. Here's another example for the October 2010 M1 turn: {noformat} http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&\ p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer|id=35591&p2Type=Date&p2Value=2010-10-27T21:00:00Z&p3Type=boolean&p3Value=false&p4Type=Date&\ p4Value=2010-q0-27T20:30:00Z&p5Type=String&p5Value=getM1-download&p6Type=String&p6Value=&delimiter=| (This returned a new DataProducerID of 41556. The ParentID_FK for this record in the DataProducer table was then changed from 35388 to 41197 and the endDate cleared to null. Also, with the change to alternating deviceIDs implemented in 2010 the deviceID_FK was changed from 1698 to 1768 for this M1 deployment. {noformat} N.B. A rotating scheme for the virtual device IDs was implemented in 2010. This permits the steps described above to be executed prior to the mooring turn during the extensive dock-side test period. h4. Enable processing for other "virtual" devices # Turn on ClockSync processing. We also use a virtual device ID for these data. Simply uncomment the line for it in the ssds.cfg file. # Turn on ISUS processing. Simply uncomment the line for it in the ssds.cfg file. (Device ID re-used from previous deployment.) h4. Make all of the child instrument deployment start times the same as the mooring start time When SSDS receives a packet from an instrument that is not currently deployed it will create a Deployment record (in the DataProducer table) with a startDate that is set to the time of the first record received. As the mooring starts up all of the instrument deployments will have different start times based on when they each first saw data. This can present problems for the data processing that follows, especially for the jobs that aggregate the microcat data into a single ZT file that is used to produce the contour temperature and salinity wind stick plots. To prevent these problems it's best to edit the startDates of the child instrument deployments so that they are all the same. This is currently most easily done through Enterprise Manager with a query like below (this is for the 201010 M1 deployment) to get all the child instrument deployments: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 41197) ORDER BY startDate {noformat} Then copy and paste the datetime string from one cell to the next. The times are GMT. h4. Cycle links to previous deployment # Edit previous.html file in /mbari/ssdsdata/deployments to add a line for the new deployment and add the end date and archive url for the just closed deployment, e.g.: {noformat}
  • 200904 2009-04-29 to 2010-04-03
  • 201004 2010-04-03 to present
    • {noformat}\\ # Add lines to DEPLOYMENTS file in dev/DPforSSDS/cimt/ for the just closed deployment, e.g.: {noformat} DStoNetCDF.pl -mooring M2 -deployment "M2 - April 2009" -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org -outputDir \ /mbari/ssdsdata/deployments -procClosed -verbose combineM.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments combineTS.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments combineAll.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments and /bin/cp /mbari/ssdsdata/deployments/m2/200904/OS_*.nc /mbari/FTP/pub/OceanSITES {noformat}\\ # Execute these lines to generate "closed deployment" data products and web pages and for submission to the OceanSITES GDAC Mike McCann (First edit: 30 October 2007, Last updated: 1 May 2009)]]>     18579771     18645285 18579770 18645283 18579768 18645290 18579775 17891557 17858813 9830663 9797895 9830664 9797896 17891561 17858817 18645312 18579797 18645301 18579786 17891563 17858819 18645299 18579784 18645300 18579785 18645305 18579790 18645306 18579791 18645303 18579788 15663982 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs and the path to the xml files to the newly deployed deviceIDs (aka ISI_IDs) and new xml directory. Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen or JEdit. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml file with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. \[Note: With the M1 - October 2009 deployment we added the individual inductive modem microcats as children of the mooring and also configured a parallel deployment of the TString that produced the same data. With the M2 - April 2010 deployment we configured the individual IM mirocrocats and did not configure a parallel TString deployment. This is a better model for the system and is much easier to configure and produces more easily consumed data by downstream processes such as combineTS.pl.\] With future deployments we will not configure TString. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /mbari/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. \\ For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed (or...as below \-rschramm 4/2010) \\ \\ {noformat} declare @myID as bigint set @myID = 31548 SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE id = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk in ( SELECT id FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL) ORDER BY id {noformat} h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSDS ingest will recognize them as new. # +Do not close (set endDates) the existing Test deployment until after the oasisToSSDS has run with the new platform deployment name+ and all new instrument deployments have been created in SSDS_Metadata. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadata, e.g.: {noformat} SELECT id, name, startDate, endDate, ParentID_FK FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') ORDER BY id DESC {noformat} Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file. # You will also need to edit the DStoNetCDF.pl script to set the new name of the M1 deployment, e.g.: {noformat} # # SSDS mooring name Deployment lookup # my %ssdsMooringDeplNames = ( M0 => 'M0', M1 => 'M1 - October 2009', M1Test => 'Test M1 - October 2009', M2 => 'M2 - April 2009', {noformat} # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. # When the mooring log message is sent to oasis with the actual time of deployment enter that as the startDate in the database.  All times (DTGs) in the SSDS_Metadata database are GMT. Make sure to set the startDate for the imctd microcat sensors too. This can be done with a query like this where you use the proper dataProduceIDs for the mooring and the imcd: {noformat} SELECT id, name, ParentID_FK, startDate, endDate, nominalDepth FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') AND (ParentID_FK = 31564) OR (ParentID_FK = 31548) ORDER BY id DESC {noformat} # Because of some bug in SSDS ingest the dataContainerTypes of the outputs from the instrument deployments do not get properly assigned the values of 'Stream'.  This needs to be fixed so that the NDBC datatransfers will work. To fix it edit the SSDS_Metadata database starting with a query like this: {noformat} SELECT ssdsdba.DataContainer.* FROM ssdsdba.DataProducer INNER JOIN ssdsdba.DataContainer ON ssdsdba.DataProducer.id = ssdsdba.DataContainer.DataProducerID_FK WHERE (ssdsdba.DataProducer.ParentID_FK = 31548) {noformat} where you use the DataProducerID for the new mooring deployment in the WHERE clause. ==> Change all of the 'File's in the dataContainerType' field to 'Stream's. h2. C. Procedures to be done after the mooring turn h4. Set up download info deployment and reporting Though not a real instrument, we configure a virtual 'dlinfo' instrument for the download scripts to attach download statistics data. We re-use the same device IDs for the M1 and M2 moorings, so it's best to configure this after the new mooring is out and all of those deployments have been closed. As the data are delivered "out of band" from the typical OASIS instruments we need to create a deployment for dlinfo instrument by hand. # The most direct way is to use the createDuplicateDeepDeployment service call. For example, to duplicate the 2009 M2 dlinfo deployment for the 2010 M2 deployment this call was executed: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer\|id=34496&p2Type=Date&p2Value=2010-04-03T22:00:00Z&p3Type=boolean&p3Value=false&p4Type=Date&p4Value=2010-04-03T23:00:00Z&p5Type=String&p5Value=getM2-download&p6Type=String&p6Value=&delimiter=\|. Of course you will need to adjust the DataProducer ID Date values for the new deployment you are creating. Here is the Key to the parameters: {noformat} Key: ---- http://localhost:8080/servlet/MetadataAccessServlet ?responseType=text &delimiter=| &objectToInvokeOn=DataProducerAccess &method=createDuplicateDeepDeployment &p1Type=DataProducer &p1Value=DataProducer|id=XXXX (XXXX is the ID of the deployment to copy) &p2Type=Date &p2Value=XXXXXX (XXXXXX is the start date of the new copy in XML format YYYY-MM-DDTHH:MM:SSZ) &p3Type=boolean &p3Value=(true|false) (this is to indicate if you want the original deployment to be closed) &p4Type=Date &p4Value=XXXXXX (XXXXXX is the end date for the original deployment (if p3Value is true)) &p5Type=String &p5Value=XXXXXX (XXXXXX is the DataProducer name of the new DataProducer) &p6Type=String &p6Value=XXXXXX (XXXXXX is the base URL to use for the new DataContainers that will be created). &delimiter=| {noformat}\\ Executing the createDuplicateDeepDeployment service call will return an ID for the new deployment. # Edit the new deployment record in the DataProducer table to adjust it's parentID_FK to be for the new mooring. While there edit times and name as appropriate. # Edit the getM? script on tsunami to use the proper device and parent ID (this will be the device ID of the torroid of the mooring deployment) and make sure that the '/oasis/bin/ssdsSubmit.pl $deviceId $parentId "$starttime_es,$endtime_es,$filesize,$rtnsts"' line in the script is configured to run. h4. Enable processing for other "virtual" devices # Turn on ClockSync processing. We also use a virtual device ID for these data. Simply uncomment the line for it in the ssds.cfg file. # Turn on ISUS processing. Simply uncomment the line for it in the ssds.cfg file. (Device ID re-used from previous deployment.) h4. Cycle links to previous deployment # Edit previous.html file in /mbari/ssdsdata/deployments to add a line for the new deployment and add the end date and archive url for the just closed deployment, e.g.: {noformat}
  • 200904 2009-04-29 to 2010-04-03
  • 201004 2010-04-03 to present
    • {noformat}\\ # Add lines to DEPLOYMENTS file in dev/DPforSSDS/cimt/ for the just closed deployment, e.g.: {noformat} DStoNetCDF.pl -mooring M2 -deployment "M2 - April 2009" -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org -outputDir /mbari/ssdsdata/deployments -procClosed -verbose combineM.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments combineTS.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments combineAll.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments and /bin/cp /mbari/ssdsdata/deployments/m2/200904/OS_*.nc /mbari/FTP/pub/OceanSITES {noformat}\\ # Execute these lines to generate "closed deployment" data products and web pages and for submission to the OceanSITES GDAC Mike McCann (First edit: 30 October 2007, Last updated: 1 May 2009)]]>     15631231     15663986 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs and the path to the xml files to the newly deployed deviceIDs (aka ISI_IDs) and new xml directory. Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen or JEdit. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml file with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. \[Note: With the M1 - October 2009 deployment we added the individual inductive modem microcats as children of the mooring and also configured a parallel deployment of the TString that produced the same data. With the M2 - April 2010 deployment we configured the individual IM mirocrocats and did not configure a parallel TString deployment. This is a better model for the system and is much easier to configure and produces more easily consumed data by downstream processes such as combineTS.pl.\] With future deployments we will not configure TString. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /mbari/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. \\ For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed (or...as below \-rschramm 4/2010) \\ \\ {noformat} declare @myID as bigint set @myID = 31548 SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE id = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk in ( SELECT id FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL) ORDER BY id {noformat} h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSDS ingest will recognize them as new. # +Do not close (set endDates) the existing Test deployment until after the oasisToSSDS has run with the new platform deployment name+ and all new instrument deployments have been created in SSDS_Metadata. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadata, e.g.: {noformat} SELECT id, name, startDate, endDate, ParentID_FK FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') ORDER BY id DESC {noformat} Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file. # You will also need to edit the DStoNetCDF.pl script to set the new name of the M1 deployment, e.g.: {noformat} # # SSDS mooring name Deployment lookup # my %ssdsMooringDeplNames = ( M0 => 'M0', M1 => 'M1 - October 2009', M1Test => 'Test M1 - October 2009', M2 => 'M2 - April 2009', {noformat} # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. # When the mooring log message is sent to oasis with the actual time of deployment enter that as the startDate in the database.  All times (DTGs) in the SSDS_Metadata database are GMT. Make sure to set the startDate for the imctd microcat sensors too. This can be done with a query like this where you use the proper dataProduceIDs for the mooring and the imcd: {noformat} SELECT id, name, ParentID_FK, startDate, endDate, nominalDepth FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') AND (ParentID_FK = 31564) OR (ParentID_FK = 31548) ORDER BY id DESC {noformat} # Because of some bug in SSDS ingest the dataContainerTypes of the outputs from the instrument deployments do not get properly assigned the values of 'Stream'.  This needs to be fixed so that the NDBC datatransfers will work. To fix it edit the SSDS_Metadata database starting with a query like this: {noformat} SELECT ssdsdba.DataContainer.* FROM ssdsdba.DataProducer INNER JOIN ssdsdba.DataContainer ON ssdsdba.DataProducer.id = ssdsdba.DataContainer.DataProducerID_FK WHERE (ssdsdba.DataProducer.ParentID_FK = 31548) {noformat} where you use the DataProducerID for the new mooring deployment in the WHERE clause. ==> Change all of the 'File's in the dataContainerType' field to 'Stream's. h2. C. Procedures to be done after the mooring turn h4. Set up download info deployment and reporting Though not a real instrument, we configure a virtual 'dlinfo' instrument for the download scripts to attach download statistics data. We re-use the same device IDs for the M1 and M2 moorings, so it's best to configure this after the new mooring is out and all of those deployments have been closed. As the data are delivered "out of band" from the typical OASIS instruments we need to create a deployment for dlinfo instrument by hand. # The most direct way is to use the createDuplicateDeepDeployment service call. For example, to duplicate the 2009 M2 dlinfo deployment for the 2010 M2 deployment this call was executed: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer\|id=*34496*&p2Type=Date&p2Value=*2010-04-03T22:00:00Z*&p3Type=boolean&p3Value=false&p4Type=Date&p4Value=*2010-04-03T23:00:00Z*&p5Type=String&p5Value=getM2-download&p6Type=String&p6Value=&delimiter=\|. Of course you will need to adjust the DataProducer ID Date values for the new deployment you are creating. Here is the Key to the parameters: {noformat} Key: ---- http://localhost:8080/servlet/MetadataAccessServlet ?responseType=text &delimiter=| &objectToInvokeOn=DataProducerAccess &method=createDuplicateDeepDeployment &p1Type=DataProducer &p1Value=DataProducer|id=XXXX (XXXX is the ID of the deployment to copy) &p2Type=Date &p2Value=XXXXXX (XXXXXX is the start date of the new copy in XML format YYYY-MM-DDTHH:MM:SSZ) &p3Type=boolean &p3Value=(true|false) (this is to indicate if you want the original deployment to be closed) &p4Type=Date &p4Value=XXXXXX (XXXXXX is the end date for the original deployment (if p3Value is true)) &p5Type=String &p5Value=XXXXXX (XXXXXX is the DataProducer name of the new DataProducer) &p6Type=String &p6Value=XXXXXX (XXXXXX is the base URL to use for the new DataContainers that will be created). &delimiter=| {noformat}\\ Executing the createDuplicateDeepDeployment service call will return an ID for the new deployment. Here's another example for the October 2010 M1 turn: {code} http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer|id=35591&p2Type=Date&p2Value=2010-10-27T21:00:00Z&p3Type=boolean&p3Value=false&p4Type=Date&p4Value=2010-q0-27T20:30:00Z&p5Type=String&p5Value=getM1-download&p6Type=String&p6Value=&delimiter=| {code} # Edit the new deployment record in the DataProducer table to adjust it's parentID_FK to be for the new mooring. While there edit times and name as appropriate. # Edit the getM? script on tsunami to use the proper device and parent ID (this will be the device ID of the torroid of the mooring deployment) and make sure that the '/oasis/bin/ssdsSubmit.pl $deviceId $parentId "$starttime_es,$endtime_es,$filesize,$rtnsts"' line in the script is configured to run. h4. Enable processing for other "virtual" devices # Turn on ClockSync processing. We also use a virtual device ID for these data. Simply uncomment the line for it in the ssds.cfg file. # Turn on ISUS processing. Simply uncomment the line for it in the ssds.cfg file. (Device ID re-used from previous deployment.) h4. Cycle links to previous deployment # Edit previous.html file in /mbari/ssdsdata/deployments to add a line for the new deployment and add the end date and archive url for the just closed deployment, e.g.: {noformat}
  • 200904 2009-04-29 to 2010-04-03
  • 201004 2010-04-03 to present
    • {noformat}\\ # Add lines to DEPLOYMENTS file in dev/DPforSSDS/cimt/ for the just closed deployment, e.g.: {noformat} DStoNetCDF.pl -mooring M2 -deployment "M2 - April 2009" -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org -outputDir /mbari/ssdsdata/deployments -procClosed -verbose combineM.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments combineTS.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments combineAll.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments and /bin/cp /mbari/ssdsdata/deployments/m2/200904/OS_*.nc /mbari/FTP/pub/OceanSITES {noformat}\\ # Execute these lines to generate "closed deployment" data products and web pages and for submission to the OceanSITES GDAC Mike McCann (First edit: 30 October 2007, Last updated: 1 May 2009)]]>     15631235     15663984 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs and the path to the xml files to the newly deployed deviceIDs (aka ISI_IDs) and new xml directory. Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen or JEdit. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml file with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. \[Note: With the M1 - October 2009 deployment we added the individual inductive modem microcats as children of the mooring and also configured a parallel deployment of the TString that produced the same data. With the M2 - April 2010 deployment we configured the individual IM mirocrocats and did not configure a parallel TString deployment. This is a better model for the system and is much easier to configure and produces more easily consumed data by downstream processes such as combineTS.pl.\] With future deployments we will not configure TString. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /mbari/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. \\ For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed (or...as below \-rschramm 4/2010) \\ \\ {noformat} declare @myID as bigint set @myID = 31548 SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE id = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk in ( SELECT id FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL) ORDER BY id {noformat} h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSDS ingest will recognize them as new. # +Do not close (set endDates) the existing Test deployment until after the oasisToSSDS has run with the new platform deployment name+ and all new instrument deployments have been created in SSDS_Metadata. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadata, e.g.: {noformat} SELECT id, name, startDate, endDate, ParentID_FK FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') ORDER BY id DESC {noformat} Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file. # You will also need to edit the DStoNetCDF.pl script to set the new name of the M1 deployment, e.g.: {noformat} # # SSDS mooring name Deployment lookup # my %ssdsMooringDeplNames = ( M0 => 'M0', M1 => 'M1 - October 2009', M1Test => 'Test M1 - October 2009', M2 => 'M2 - April 2009', {noformat} # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. # When the mooring log message is sent to oasis with the actual time of deployment enter that as the startDate in the database.  All times (DTGs) in the SSDS_Metadata database are GMT. Make sure to set the startDate for the imctd microcat sensors too. This can be done with a query like this where you use the proper dataProduceIDs for the mooring and the imcd: {noformat} SELECT id, name, ParentID_FK, startDate, endDate, nominalDepth FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') AND (ParentID_FK = 31564) OR (ParentID_FK = 31548) ORDER BY id DESC {noformat} # Because of some bug in SSDS ingest the dataContainerTypes of the outputs from the instrument deployments do not get properly assigned the values of 'Stream'.  This needs to be fixed so that the NDBC datatransfers will work. To fix it edit the SSDS_Metadata database starting with a query like this: {noformat} SELECT ssdsdba.DataContainer.* FROM ssdsdba.DataProducer INNER JOIN ssdsdba.DataContainer ON ssdsdba.DataProducer.id = ssdsdba.DataContainer.DataProducerID_FK WHERE (ssdsdba.DataProducer.ParentID_FK = 31548) {noformat} where you use the DataProducerID for the new mooring deployment in the WHERE clause. ==> Change all of the 'File's in the dataContainerType' field to 'Stream's. h2. C. Procedures to be done after the mooring turn h4. Set up download info deployment and reporting Though not a real instrument, we configure a virtual 'dlinfo' instrument for the download scripts to attach download statistics data. We re-use the same device IDs for the M1 and M2 moorings, so it's best to configure this after the new mooring is out and all of those deployments have been closed. As the data are delivered "out of band" from the typical OASIS instruments we need to create a deployment for dlinfo instrument by hand. # The most direct way is to use the createDuplicateDeepDeployment service call. For example, to duplicate the 2009 M2 dlinfo deployment for the 2010 M2 deployment this call was executed: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer\|id=34496&p2Type=Date&p2Value=2010-04-03T22:00:00Z&p3Type=boolean&p3Value=false&p4Type=Date&p4Value=2010-04-03T23:00:00Z&p5Type=String&p5Value=getM2-download&p6Type=String&p6Value=&delimiter=\|. Of course you will need to adjust the DataProducer ID Date values for the new deployment you are creating. Here is the Key to the parameters: {noformat} Key: ---- http://localhost:8080/servlet/MetadataAccessServlet ?responseType=text &delimiter=| &objectToInvokeOn=DataProducerAccess &method=createDuplicateDeepDeployment &p1Type=DataProducer &p1Value=DataProducer|id=XXXX (XXXX is the ID of the deployment to copy) &p2Type=Date &p2Value=XXXXXX (XXXXXX is the start date of the new copy in XML format YYYY-MM-DDTHH:MM:SSZ) &p3Type=boolean &p3Value=(true|false) (this is to indicate if you want the original deployment to be closed) &p4Type=Date &p4Value=XXXXXX (XXXXXX is the end date for the original deployment (if p3Value is true)) &p5Type=String &p5Value=XXXXXX (XXXXXX is the DataProducer name of the new DataProducer) &p6Type=String &p6Value=XXXXXX (XXXXXX is the base URL to use for the new DataContainers that will be created). &delimiter=| {noformat}\\ Executing the createDuplicateDeepDeployment service call will return an ID for the new deployment. Here's another example for the October 2010 M1 turn: {noformat} http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer|id=35591&p2Type=Date&p2Value=2010-10-27T21:00:00Z&p3Type=boolean&p3Value=false&p4Type=Date&p4Value=2010-q0-27T20:30:00Z&p5Type=String&p5Value=getM1-download&p6Type=String&p6Value=&delimiter=| {noformat} # Edit the new deployment record in the DataProducer table to adjust it's parentID_FK to be for the new mooring. While there edit times and name as appropriate. # Edit the getM? script on tsunami to use the proper device and parent ID (this will be the device ID of the torroid of the mooring deployment) and make sure that the '/oasis/bin/ssdsSubmit.pl $deviceId $parentId "$starttime_es,$endtime_es,$filesize,$rtnsts"' line in the script is configured to run. h4. Enable processing for other "virtual" devices # Turn on ClockSync processing. We also use a virtual device ID for these data. Simply uncomment the line for it in the ssds.cfg file. # Turn on ISUS processing. Simply uncomment the line for it in the ssds.cfg file. (Device ID re-used from previous deployment.) h4. Cycle links to previous deployment # Edit previous.html file in /mbari/ssdsdata/deployments to add a line for the new deployment and add the end date and archive url for the just closed deployment, e.g.: {noformat}
  • 200904 2009-04-29 to 2010-04-03
  • 201004 2010-04-03 to present
    • {noformat}\\ # Add lines to DEPLOYMENTS file in dev/DPforSSDS/cimt/ for the just closed deployment, e.g.: {noformat} DStoNetCDF.pl -mooring M2 -deployment "M2 - April 2009" -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org -outputDir /mbari/ssdsdata/deployments -procClosed -verbose combineM.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments combineTS.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments combineAll.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments and /bin/cp /mbari/ssdsdata/deployments/m2/200904/OS_*.nc /mbari/FTP/pub/OceanSITES {noformat}\\ # Execute these lines to generate "closed deployment" data products and web pages and for submission to the OceanSITES GDAC Mike McCann (First edit: 30 October 2007, Last updated: 1 May 2009)]]>     15631233     15663990 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs and the path to the xml files to the newly deployed deviceIDs (aka ISI_IDs) and new xml directory. Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen or JEdit. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml file with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. \[Note: With the M1 - October 2009 deployment we added the individual inductive modem microcats as children of the mooring and also configured a parallel deployment of the TString that produced the same data. With the M2 - April 2010 deployment we configured the individual IM mirocrocats and did not configure a parallel TString deployment. This is a better model for the system and is much easier to configure and produces more easily consumed data by downstream processes such as combineTS.pl.\] With future deployments we will not configure TString. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /mbari/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. \\ For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed (or...as below \-rschramm 4/2010) \\ \\ {noformat} declare @myID as bigint set @myID = 31548 SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE id = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk in ( SELECT id FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL) ORDER BY id {noformat} h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSDS ingest will recognize them as new. # +Do not close (set endDates) the existing Test deployment until after the oasisToSSDS has run with the new platform deployment name+ and all new instrument deployments have been created in SSDS_Metadata. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadata, e.g.: {noformat} SELECT id, name, startDate, endDate, ParentID_FK FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') ORDER BY id DESC {noformat} Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file. # You will also need to edit the DStoNetCDF.pl script to set the new name of the M1 deployment, e.g.: {noformat} # # SSDS mooring name Deployment lookup # my %ssdsMooringDeplNames = ( M0 => 'M0', M1 => 'M1 - October 2009', M1Test => 'Test M1 - October 2009', M2 => 'M2 - April 2009', {noformat} # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. # When the mooring log message is sent to oasis with the actual time of deployment enter that as the startDate in the database.  All times (DTGs) in the SSDS_Metadata database are GMT. Make sure to set the startDate for the imctd microcat sensors too. This can be done with a query like this where you use the proper dataProduceIDs for the mooring and the imcd: {noformat} SELECT id, name, ParentID_FK, startDate, endDate, nominalDepth FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') AND (ParentID_FK = 31564) OR (ParentID_FK = 31548) ORDER BY id DESC {noformat} # Because of some bug in SSDS ingest the dataContainerTypes of the outputs from the instrument deployments do not get properly assigned the values of 'Stream'.  This needs to be fixed so that the NDBC datatransfers will work. To fix it edit the SSDS_Metadata database starting with a query like this: {noformat} SELECT ssdsdba.DataContainer.* FROM ssdsdba.DataProducer INNER JOIN ssdsdba.DataContainer ON ssdsdba.DataProducer.id = ssdsdba.DataContainer.DataProducerID_FK WHERE (ssdsdba.DataProducer.ParentID_FK = 31548) {noformat} where you use the DataProducerID for the new mooring deployment in the WHERE clause. ==> Change all of the 'File's in the dataContainerType' field to 'Stream's. h2. C. Procedures to be done after the mooring turn h4. Set up download info deployment and reporting Though not a real instrument, we configure a virtual 'dlinfo' instrument for the download scripts to attach download statistics data. We re-use the same device IDs for the M1 and M2 moorings, so it's best to configure this after the new mooring is out and all of those deployments have been closed. As the data are delivered "out of band" from the typical OASIS instruments we need to create a deployment for dlinfo instrument by hand. # The most direct way is to use the createDuplicateDeepDeployment service call. For example, to duplicate the 2009 M2 dlinfo deployment for the 2010 M2 deployment this call was executed: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer\|id=*34496*&p2Type=Date&p2Value=*2010-04-03T22:00:00Z*&p3Type=boolean&p3Value=false&p4Type=Date&p4Value=*2010-04-03T23:00:00Z*&p5Type=String&p5Value=getM2-download&p6Type=String&p6Value=&delimiter=\|. Of course you will need to adjust the DataProducer ID Date values for the new deployment you are creating. Here is the Key to the parameters: {noformat} Key: ---- http://localhost:8080/servlet/MetadataAccessServlet ?responseType=text &delimiter=| &objectToInvokeOn=DataProducerAccess &method=createDuplicateDeepDeployment &p1Type=DataProducer &p1Value=DataProducer|id=XXXX (XXXX is the ID of the deployment to copy) &p2Type=Date &p2Value=XXXXXX (XXXXXX is the start date of the new copy in XML format YYYY-MM-DDTHH:MM:SSZ) &p3Type=boolean &p3Value=(true|false) (this is to indicate if you want the original deployment to be closed) &p4Type=Date &p4Value=XXXXXX (XXXXXX is the end date for the original deployment (if p3Value is true)) &p5Type=String &p5Value=XXXXXX (XXXXXX is the DataProducer name of the new DataProducer) &p6Type=String &p6Value=XXXXXX (XXXXXX is the base URL to use for the new DataContainers that will be created). &delimiter=| {noformat}\\ Executing the createDuplicateDeepDeployment service call will return an ID for the new deployment. Here's another example for the October 2010 M1 turn: {code} http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&\ p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer|id=35591&p2Type=Date&p2Value=2010-10-27T21:00:00Z&p3Type=boolean&p3Value=false&p4Type=Date&\ p4Value=2010-q0-27T20:30:00Z&p5Type=String&p5Value=getM1-download&p6Type=String&p6Value=&delimiter=| This returned a new DataProducerID of 41556. The ParentID_FK for this record in the DataProducer table was then changed from 35388 to 41197. {code} # Edit the new deployment record in the DataProducer table to adjust it's parentID_FK to be for the new mooring. While there edit times and name as appropriate. # Edit the getM? script on tsunami to use the proper device and parent ID (this will be the device ID of the torroid of the mooring deployment) and make sure that the '/oasis/bin/ssdsSubmit.pl $deviceId $parentId "$starttime_es,$endtime_es,$filesize,$rtnsts"' line in the script is configured to run. h4. Enable processing for other "virtual" devices # Turn on ClockSync processing. We also use a virtual device ID for these data. Simply uncomment the line for it in the ssds.cfg file. # Turn on ISUS processing. Simply uncomment the line for it in the ssds.cfg file. (Device ID re-used from previous deployment.) h4. Cycle links to previous deployment # Edit previous.html file in /mbari/ssdsdata/deployments to add a line for the new deployment and add the end date and archive url for the just closed deployment, e.g.: {noformat}
  • 200904 2009-04-29 to 2010-04-03
  • 201004 2010-04-03 to present
    • {noformat}\\ # Add lines to DEPLOYMENTS file in dev/DPforSSDS/cimt/ for the just closed deployment, e.g.: {noformat} DStoNetCDF.pl -mooring M2 -deployment "M2 - April 2009" -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org -outputDir /mbari/ssdsdata/deployments -procClosed -verbose combineM.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments combineTS.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments combineAll.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments and /bin/cp /mbari/ssdsdata/deployments/m2/200904/OS_*.nc /mbari/FTP/pub/OceanSITES {noformat}\\ # Execute these lines to generate "closed deployment" data products and web pages and for submission to the OceanSITES GDAC Mike McCann (First edit: 30 October 2007, Last updated: 1 May 2009)]]>     15631239     15663988 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs and the path to the xml files to the newly deployed deviceIDs (aka ISI_IDs) and new xml directory. Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen or JEdit. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml file with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. \[Note: With the M1 - October 2009 deployment we added the individual inductive modem microcats as children of the mooring and also configured a parallel deployment of the TString that produced the same data. With the M2 - April 2010 deployment we configured the individual IM mirocrocats and did not configure a parallel TString deployment. This is a better model for the system and is much easier to configure and produces more easily consumed data by downstream processes such as combineTS.pl.\] With future deployments we will not configure TString. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /mbari/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. \\ For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed (or...as below \-rschramm 4/2010) \\ \\ {noformat} declare @myID as bigint set @myID = 31548 SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE id = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk in ( SELECT id FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL) ORDER BY id {noformat} h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSDS ingest will recognize them as new. # +Do not close (set endDates) the existing Test deployment until after the oasisToSSDS has run with the new platform deployment name+ and all new instrument deployments have been created in SSDS_Metadata. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadata, e.g.: {noformat} SELECT id, name, startDate, endDate, ParentID_FK FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') ORDER BY id DESC {noformat} Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file. # You will also need to edit the DStoNetCDF.pl script to set the new name of the M1 deployment, e.g.: {noformat} # # SSDS mooring name Deployment lookup # my %ssdsMooringDeplNames = ( M0 => 'M0', M1 => 'M1 - October 2009', M1Test => 'Test M1 - October 2009', M2 => 'M2 - April 2009', {noformat} # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. # When the mooring log message is sent to oasis with the actual time of deployment enter that as the startDate in the database.  All times (DTGs) in the SSDS_Metadata database are GMT. Make sure to set the startDate for the imctd microcat sensors too. This can be done with a query like this where you use the proper dataProduceIDs for the mooring and the imcd: {noformat} SELECT id, name, ParentID_FK, startDate, endDate, nominalDepth FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') AND (ParentID_FK = 31564) OR (ParentID_FK = 31548) ORDER BY id DESC {noformat} # Because of some bug in SSDS ingest the dataContainerTypes of the outputs from the instrument deployments do not get properly assigned the values of 'Stream'.  This needs to be fixed so that the NDBC datatransfers will work. To fix it edit the SSDS_Metadata database starting with a query like this: {noformat} SELECT ssdsdba.DataContainer.* FROM ssdsdba.DataProducer INNER JOIN ssdsdba.DataContainer ON ssdsdba.DataProducer.id = ssdsdba.DataContainer.DataProducerID_FK WHERE (ssdsdba.DataProducer.ParentID_FK = 31548) {noformat} where you use the DataProducerID for the new mooring deployment in the WHERE clause. ==> Change all of the 'File's in the dataContainerType' field to 'Stream's. h2. C. Procedures to be done after the mooring turn h4. Set up download info deployment and reporting Though not a real instrument, we configure a virtual 'dlinfo' instrument for the download scripts to attach download statistics data. We re-use the same device IDs for the M1 and M2 moorings, so it's best to configure this after the new mooring is out and all of those deployments have been closed. As the data are delivered "out of band" from the typical OASIS instruments we need to create a deployment for dlinfo instrument by hand. # The most direct way is to use the createDuplicateDeepDeployment service call. For example, to duplicate the 2009 M2 dlinfo deployment for the 2010 M2 deployment this call was executed: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer\|id=*34496*&p2Type=Date&p2Value=*2010-04-03T22:00:00Z*&p3Type=boolean&p3Value=false&p4Type=Date&p4Value=*2010-04-03T23:00:00Z*&p5Type=String&p5Value=getM2-download&p6Type=String&p6Value=&delimiter=\|. Of course you will need to adjust the DataProducer ID Date values for the new deployment you are creating. Here is the Key to the parameters: {noformat} Key: ---- http://localhost:8080/servlet/MetadataAccessServlet ?responseType=text &delimiter=| &objectToInvokeOn=DataProducerAccess &method=createDuplicateDeepDeployment &p1Type=DataProducer &p1Value=DataProducer|id=XXXX (XXXX is the ID of the deployment to copy) &p2Type=Date &p2Value=XXXXXX (XXXXXX is the start date of the new copy in XML format YYYY-MM-DDTHH:MM:SSZ) &p3Type=boolean &p3Value=(true|false) (this is to indicate if you want the original deployment to be closed) &p4Type=Date &p4Value=XXXXXX (XXXXXX is the end date for the original deployment (if p3Value is true)) &p5Type=String &p5Value=XXXXXX (XXXXXX is the DataProducer name of the new DataProducer) &p6Type=String &p6Value=XXXXXX (XXXXXX is the base URL to use for the new DataContainers that will be created). &delimiter=| {noformat}\\ Executing the createDuplicateDeepDeployment service call will return an ID for the new deployment. Here's another example for the October 2010 M1 turn: {code} http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&\ p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer|id=35591&p2Type=Date&p2Value=2010-10-27T21:00:00Z&p3Type=boolean&p3Value=false&p4Type=Date&\ p4Value=2010-q0-27T20:30:00Z&p5Type=String&p5Value=getM1-download&p6Type=String&p6Value=&delimiter=| {code} # Edit the new deployment record in the DataProducer table to adjust it's parentID_FK to be for the new mooring. While there edit times and name as appropriate. # Edit the getM? script on tsunami to use the proper device and parent ID (this will be the device ID of the torroid of the mooring deployment) and make sure that the '/oasis/bin/ssdsSubmit.pl $deviceId $parentId "$starttime_es,$endtime_es,$filesize,$rtnsts"' line in the script is configured to run. h4. Enable processing for other "virtual" devices # Turn on ClockSync processing. We also use a virtual device ID for these data. Simply uncomment the line for it in the ssds.cfg file. # Turn on ISUS processing. Simply uncomment the line for it in the ssds.cfg file. (Device ID re-used from previous deployment.) h4. Cycle links to previous deployment # Edit previous.html file in /mbari/ssdsdata/deployments to add a line for the new deployment and add the end date and archive url for the just closed deployment, e.g.: {noformat}
  • 200904 2009-04-29 to 2010-04-03
  • 201004 2010-04-03 to present
    • {noformat}\\ # Add lines to DEPLOYMENTS file in dev/DPforSSDS/cimt/ for the just closed deployment, e.g.: {noformat} DStoNetCDF.pl -mooring M2 -deployment "M2 - April 2009" -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org -outputDir /mbari/ssdsdata/deployments -procClosed -verbose combineM.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments combineTS.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments combineAll.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments and /bin/cp /mbari/ssdsdata/deployments/m2/200904/OS_*.nc /mbari/FTP/pub/OceanSITES {noformat}\\ # Execute these lines to generate "closed deployment" data products and web pages and for submission to the OceanSITES GDAC Mike McCann (First edit: 30 October 2007, Last updated: 1 May 2009)]]>     15631237     15663991 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs and the path to the xml files to the newly deployed deviceIDs (aka ISI_IDs) and new xml directory. Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen or JEdit. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml file with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. \[Note: With the M1 - October 2009 deployment we added the individual inductive modem microcats as children of the mooring and also configured a parallel deployment of the TString that produced the same data. With the M2 - April 2010 deployment we configured the individual IM mirocrocats and did not configure a parallel TString deployment. This is a better model for the system and is much easier to configure and produces more easily consumed data by downstream processes such as combineTS.pl.\] With future deployments we will not configure TString. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /mbari/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. \\ For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed (or...as below \-rschramm 4/2010) \\ \\ {noformat} declare @myID as bigint set @myID = 31548 SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE id = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk in ( SELECT id FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL) ORDER BY id {noformat} h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSDS ingest will recognize them as new. # +Do not close (set endDates) the existing Test deployment until after the oasisToSSDS has run with the new platform deployment name+ and all new instrument deployments have been created in SSDS_Metadata. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadata, e.g.: {noformat} SELECT id, name, startDate, endDate, ParentID_FK FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') ORDER BY id DESC {noformat} Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file. # You will also need to edit the DStoNetCDF.pl script to set the new name of the M1 deployment, e.g.: {noformat} # # SSDS mooring name Deployment lookup # my %ssdsMooringDeplNames = ( M0 => 'M0', M1 => 'M1 - October 2009', M1Test => 'Test M1 - October 2009', M2 => 'M2 - April 2009', {noformat} # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. # When the mooring log message is sent to oasis with the actual time of deployment enter that as the startDate in the database.  All times (DTGs) in the SSDS_Metadata database are GMT. Make sure to set the startDate for the imctd microcat sensors too. This can be done with a query like this where you use the proper dataProduceIDs for the mooring and the imcd: {noformat} SELECT id, name, ParentID_FK, startDate, endDate, nominalDepth FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') AND (ParentID_FK = 31564) OR (ParentID_FK = 31548) ORDER BY id DESC {noformat} # Because of some bug in SSDS ingest the dataContainerTypes of the outputs from the instrument deployments do not get properly assigned the values of 'Stream'.  This needs to be fixed so that the NDBC datatransfers will work. To fix it edit the SSDS_Metadata database starting with a query like this: {noformat} SELECT ssdsdba.DataContainer.* FROM ssdsdba.DataProducer INNER JOIN ssdsdba.DataContainer ON ssdsdba.DataProducer.id = ssdsdba.DataContainer.DataProducerID_FK WHERE (ssdsdba.DataProducer.ParentID_FK = 31548) {noformat} where you use the DataProducerID for the new mooring deployment in the WHERE clause. ==> Change all of the 'File's in the dataContainerType' field to 'Stream's. h2. C. Procedures to be done after the mooring turn h4. Set up download info deployment and reporting Though not a real instrument, we configure a virtual 'dlinfo' instrument for the download scripts to attach download statistics data. We re-use the same device IDs for the M1 and M2 moorings, so it's best to configure this after the new mooring is out and all of those deployments have been closed. As the data are delivered "out of band" from the typical OASIS instruments we need to create a deployment for dlinfo instrument by hand. # The most direct way is to use the createDuplicateDeepDeployment service call. For example, to duplicate the 2009 M2 dlinfo deployment for the 2010 M2 deployment this call was executed: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer\|id=*34496*&p2Type=Date&p2Value=*2010-04-03T22:00:00Z*&p3Type=boolean&p3Value=false&p4Type=Date&p4Value=*2010-04-03T23:00:00Z*&p5Type=String&p5Value=getM2-download&p6Type=String&p6Value=&delimiter=\|. Of course you will need to adjust the DataProducer ID Date values for the new deployment you are creating. Here is the Key to the parameters: {noformat} Key: ---- http://localhost:8080/servlet/MetadataAccessServlet ?responseType=text &delimiter=| &objectToInvokeOn=DataProducerAccess &method=createDuplicateDeepDeployment &p1Type=DataProducer &p1Value=DataProducer|id=XXXX (XXXX is the ID of the deployment to copy) &p2Type=Date &p2Value=XXXXXX (XXXXXX is the start date of the new copy in XML format YYYY-MM-DDTHH:MM:SSZ) &p3Type=boolean &p3Value=(true|false) (this is to indicate if you want the original deployment to be closed) &p4Type=Date &p4Value=XXXXXX (XXXXXX is the end date for the original deployment (if p3Value is true)) &p5Type=String &p5Value=XXXXXX (XXXXXX is the DataProducer name of the new DataProducer) &p6Type=String &p6Value=XXXXXX (XXXXXX is the base URL to use for the new DataContainers that will be created). &delimiter=| {noformat}\\ Executing the createDuplicateDeepDeployment service call will return an ID for the new deployment. # Edit the new deployment record in the DataProducer table to adjust it's parentID_FK to be for the new mooring. While there edit times and name as appropriate. # Edit the getM? script on tsunami to use the proper device and parent ID (this will be the device ID of the torroid of the mooring deployment) and make sure that the '/oasis/bin/ssdsSubmit.pl $deviceId $parentId "$starttime_es,$endtime_es,$filesize,$rtnsts"' line in the script is configured to run. Here's another example for the October 2010 M1 turn: {code} http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&\ p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer|id=35591&p2Type=Date&p2Value=2010-10-27T21:00:00Z&p3Type=boolean&p3Value=false&p4Type=Date&\ p4Value=2010-q0-27T20:30:00Z&p5Type=String&p5Value=getM1-download&p6Type=String&p6Value=&delimiter=| (This returned a new DataProducerID of 41556. The ParentID_FK for this record in the DataProducer table was then changed from 35388 to 41197.) {code} h4. Enable processing for other "virtual" devices # Turn on ClockSync processing. We also use a virtual device ID for these data. Simply uncomment the line for it in the ssds.cfg file. # Turn on ISUS processing. Simply uncomment the line for it in the ssds.cfg file. (Device ID re-used from previous deployment.) h4. Cycle links to previous deployment # Edit previous.html file in /mbari/ssdsdata/deployments to add a line for the new deployment and add the end date and archive url for the just closed deployment, e.g.: {noformat}
  • 200904 2009-04-29 to 2010-04-03
  • 201004 2010-04-03 to present
    • {noformat}\\ # Add lines to DEPLOYMENTS file in dev/DPforSSDS/cimt/ for the just closed deployment, e.g.: {noformat} DStoNetCDF.pl -mooring M2 -deployment "M2 - April 2009" -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org -outputDir /mbari/ssdsdata/deployments -procClosed -verbose combineM.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments combineTS.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments combineAll.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments and /bin/cp /mbari/ssdsdata/deployments/m2/200904/OS_*.nc /mbari/FTP/pub/OceanSITES {noformat}\\ # Execute these lines to generate "closed deployment" data products and web pages and for submission to the OceanSITES GDAC Mike McCann (First edit: 30 October 2007, Last updated: 1 May 2009)]]>     15631240     19138271 19038486 2012-04-16 13:52:36.423 2012-04-16 13:52:36.423 2195654 27060) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: <251 elvis.shore.mbari.org /u/ssdsadmin> dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02 \- or wait for it to run with the hourly download on tsunami. #   #   #   Mike McCann (30 October 2007)]]> 2162888 15663997 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs and the path to the xml files to the newly deployed deviceIDs (aka ISI_IDs) and new xml directory. Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen or JEdit. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml file with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. \[Note: With the M1 - October 2009 deployment we added the individual inductive modem microcats as children of the mooring and also configured a parallel deployment of the TString that produced the same data. With the M2 - April 2010 deployment we configured the individual IM mirocrocats and did not configure a parallel TString deployment. This is a better model for the system and is much easier to configure and produces more easily consumed data by downstream processes such as combineTS.pl.\] With future deployments we will not configure TString. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /mbari/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. \\ For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed (or...as below \-rschramm 4/2010) \\ \\ {noformat} declare @myID as bigint set @myID = 31548 SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE id = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk in ( SELECT id FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL) ORDER BY id {noformat} h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSDS ingest will recognize them as new. # +Do not close (set endDates) the existing Test deployment until after the oasisToSSDS has run with the new platform deployment name+ and all new instrument deployments have been created in SSDS_Metadata. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadata, e.g.: {noformat} SELECT id, name, startDate, endDate, ParentID_FK FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') ORDER BY id DESC {noformat} Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file. # You will also need to edit the DStoNetCDF.pl script to set the new name of the M1 deployment, e.g.: {noformat} # # SSDS mooring name Deployment lookup # my %ssdsMooringDeplNames = ( M0 => 'M0', M1 => 'M1 - October 2009', M1Test => 'Test M1 - October 2009', M2 => 'M2 - April 2009', {noformat} # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. # When the mooring log message is sent to oasis with the actual time of deployment enter that as the startDate in the database.  All times (DTGs) in the SSDS_Metadata database are GMT. Make sure to set the startDate for the imctd microcat sensors too. This can be done with a query like this where you use the proper dataProduceIDs for the mooring and the imcd: {noformat} SELECT id, name, ParentID_FK, startDate, endDate, nominalDepth FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') AND (ParentID_FK = 31564) OR (ParentID_FK = 31548) ORDER BY id DESC {noformat} # Because of some bug in SSDS ingest the dataContainerTypes of the outputs from the instrument deployments do not get properly assigned the values of 'Stream'.  This needs to be fixed so that the NDBC datatransfers will work. To fix it edit the SSDS_Metadata database starting with a query like this: {noformat} SELECT ssdsdba.DataContainer.* FROM ssdsdba.DataProducer INNER JOIN ssdsdba.DataContainer ON ssdsdba.DataProducer.id = ssdsdba.DataContainer.DataProducerID_FK WHERE (ssdsdba.DataProducer.ParentID_FK = 31548) {noformat} where you use the DataProducerID for the new mooring deployment in the WHERE clause. ==> Change all of the 'File's in the dataContainerType' field to 'Stream's. h2. C. Procedures to be done after the mooring turn h4. Set up download info deployment and reporting Though not a real instrument, we configure a virtual 'dlinfo' instrument for the download scripts to attach download statistics data. We re-use the same device IDs for the M1 and M2 moorings, so it's best to configure this after the new mooring is out and all of those deployments have been closed. As the data are delivered "out of band" from the typical OASIS instruments we need to create a deployment for dlinfo instrument by hand. # The most direct way is to use the createDuplicateDeepDeployment service call. For example, to duplicate the 2009 M2 dlinfo deployment for the 2010 M2 deployment this call was executed: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer\|id=*34496*&p2Type=Date&p2Value=*2010-04-03T22:00:00Z*&p3Type=boolean&p3Value=false&p4Type=Date&p4Value=*2010-04-03T23:00:00Z*&p5Type=String&p5Value=getM2-download&p6Type=String&p6Value=&delimiter=\|. Of course you will need to adjust the DataProducer ID Date values for the new deployment you are creating. Here is the Key to the parameters: {noformat} Key: ---- http://localhost:8080/servlet/MetadataAccessServlet ?responseType=text &delimiter=| &objectToInvokeOn=DataProducerAccess &method=createDuplicateDeepDeployment &p1Type=DataProducer &p1Value=DataProducer|id=XXXX (XXXX is the ID of the deployment to copy) &p2Type=Date &p2Value=XXXXXX (XXXXXX is the start date of the new copy in XML format YYYY-MM-DDTHH:MM:SSZ) &p3Type=boolean &p3Value=(true|false) (this is to indicate if you want the original deployment to be closed) &p4Type=Date &p4Value=XXXXXX (XXXXXX is the end date for the original deployment (if p3Value is true)) &p5Type=String &p5Value=XXXXXX (XXXXXX is the DataProducer name of the new DataProducer) &p6Type=String &p6Value=XXXXXX (XXXXXX is the base URL to use for the new DataContainers that will be created). &delimiter=| {noformat}\\ Executing the createDuplicateDeepDeployment service call will return an ID for the new deployment. # Edit the new deployment record in the DataProducer table to adjust it's parentID_FK to be for the new mooring. While there edit times and name as appropriate. # Edit the getM? script on tsunami to use the proper device and parent ID (this will be the device ID of the torroid of the mooring deployment) and make sure that the '/oasis/bin/ssdsSubmit.pl $deviceId $parentId "$starttime_es,$endtime_es,$filesize,$rtnsts"' line in the script is configured to run. Here's another example for the October 2010 M1 turn: {noformat} http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&\ p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer|id=35591&p2Type=Date&p2Value=2010-10-27T21:00:00Z&p3Type=boolean&p3Value=false&p4Type=Date&\ p4Value=2010-q0-27T20:30:00Z&p5Type=String&p5Value=getM1-download&p6Type=String&p6Value=&delimiter=| (This returned a new DataProducerID of 41556. The ParentID_FK for this record in the DataProducer table was then changed from 35388 to 41197.) {noformat} h4. Enable processing for other "virtual" devices # Turn on ClockSync processing. We also use a virtual device ID for these data. Simply uncomment the line for it in the ssds.cfg file. # Turn on ISUS processing. Simply uncomment the line for it in the ssds.cfg file. (Device ID re-used from previous deployment.) h4. Cycle links to previous deployment # Edit previous.html file in /mbari/ssdsdata/deployments to add a line for the new deployment and add the end date and archive url for the just closed deployment, e.g.: {noformat}
  • 200904 2009-04-29 to 2010-04-03
  • 201004 2010-04-03 to present
    • {noformat}\\ # Add lines to DEPLOYMENTS file in dev/DPforSSDS/cimt/ for the just closed deployment, e.g.: {noformat} DStoNetCDF.pl -mooring M2 -deployment "M2 - April 2009" -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org -outputDir /mbari/ssdsdata/deployments -procClosed -verbose combineM.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments combineTS.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments combineAll.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments and /bin/cp /mbari/ssdsdata/deployments/m2/200904/OS_*.nc /mbari/FTP/pub/OceanSITES {noformat}\\ # Execute these lines to generate "closed deployment" data products and web pages and for submission to the OceanSITES GDAC Mike McCann (First edit: 30 October 2007, Last updated: 1 May 2009)]]>     15631247     3670571 3637821 6226331 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. # Edit the mooring .cfg file and changed the instrument deviceIDs to the newly deployed deviceIDs (aka ISI_IDs). Refer to email from OSG for the correct numbers, e.g.: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application, e.g.: {noformat} {noformat} Remove any time or location information and check back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen. Copy the files into the _yyyy_/xml subdirectory. Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file; if you include other attributes they will overwrite what it in the database. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml wile with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp]by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /hosts/tornado_vol0/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. ##   h2. B. Procedure for doing the actual production mooring turn So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. The steps to do this are outlined below. \[As of November 2007, there are several issues with making this a fool-proof set of instructions; therefore, for now they will serve as documentation for tasks  in the 2008 SSDS Hardening project.\] \\ # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id > 27060) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the instrument swap procedure and doing a deep delete on the wrong instrument deployment. If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: <251 elvis.shore.mbari.org /u/ssdsadmin> dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02 \- or wait for it to run with the hourly download on tsunami. #   #   #   Mike McCann (30 October 2007)]]> 6259114 3670573 3637823 9404851 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. # Edit the mooring .cfg file and changed the instrument deviceIDs to the newly deployed deviceIDs (aka ISI_IDs). Refer to email from OSG for the correct numbers, e.g.: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen. Copy the files into the _yyyy_/xml subdirectory. Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file; if you include other attributes they will overwrite what it in the database. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml wile with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /hosts/tornado_vol0/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed. h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSD ingest will recognize them as new. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /hosts/tornado_vol0/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query like this where the DataProducer.id is >= the id of the new platform, e.g.: {noformat} {noformat} # ;sd;s #   #   # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. #   # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DPLOYMENTS file. # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. Mike McCann (First edit: 30 October 2007, Last updated: 8 October 2008)]]> 9372106 3670565 3637815 18579787 18645302 2 2007-10-03 12:37:04.963 2007-10-03 13:41:39.093 1835089 3670567 3637817 6226328 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. # Edit the mooring .cfg file and changed the instrument deviceIDs to the newly deployed deviceIDs (aka ISI_IDs). Refer to email from OSG for the correct numbers, e.g.: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the platform deployment XML file; its Deployment element should contain name and startDate attributes so that the parent deployment may be found in the Explorer application, e.g.: {noformat} {noformat} Remove any time or location information and check back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen. Copy the files into the _yyyy_/xml subdirectory. Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file; if you include other attributes they will overwrite what it in the database. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml wile with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. h2. B. Procedure for doing the actual production mooring turn So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. The steps to do this are outlined below. \[As of November 2007, there are several issues with making this a fool-proof set of instructions; therefore, for now they will serve as documentation for tasks  in the 2008 SSDS Hardening project.\] \\ # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id > 27060) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the instrument swap procedure and doing a deep delete on the wrong instrument deployment. If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: <251 elvis.shore.mbari.org /u/ssdsadmin> dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02 \- or wait for it to run with the hourly download on tsunami. #   #   #   Mike McCann (30 October 2007)]]> 6259111 3670569 3637819 2195691 27060) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: <251 elvis.shore.mbari.org /u/ssdsadmin> dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02 \- or wait for it to run with the hourly download on tsunami. #   #   #   Mike McCann (30 October 2007)]]> 2162925 3179959 3114488 2195692 27060) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: <251 elvis.shore.mbari.org /u/ssdsadmin> dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02 \- or wait for it to run with the hourly download on tsunami. #   #   #   Mike McCann (30 October 2007)]]> 2162926 2195694 27060) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: <251 elvis.shore.mbari.org /u/ssdsadmin> dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02 \- or wait for it to run with the hourly download on tsunami. #   #   #   Mike McCann (30 October 2007)]]> 2162928 2195695 27060) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the instrument swap procedure and doing a deep delete on the wrong instrument deployment. If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: <251 elvis.shore.mbari.org /u/ssdsadmin> dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02 \- or wait for it to run with the hourly download on tsunami. #   #   #   Mike McCann (30 October 2007)]]> 2162929 3179968 ) on the wrong instrument deployment. # Test by ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. h2. B. Procedure for doing the actual production mooring turn So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. The steps to do this are outlined below. \[As of November 2007, there are several issues with making this a fool-proof set of instructions; therefore, for now they will serve as documentation for tasks  in the 2008  SSDS Hardening project.\] \\ # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id > 27060) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the instrument swap procedure and doing a deep delete on the wrong instrument deployment. If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: <251 elvis.shore.mbari.org /u/ssdsadmin> dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02 \- or wait for it to run with the hourly download on tsunami. #   #   #   Mike McCann (30 October 2007)]]> 3114498 6226309 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. # Edit the mooring .cfg file and changed the instrument deviceIDs to the newly deployed deviceIDs (aka ISI_IDs). Refer to email from OSG for the correct numbers, e.g.: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files. The one exception is the platform deployment XML file; its Deployment element should contain name and startDate attributes so that the parent deployment may be found in the Explorer application, e.g.: {noformat} {noformat} Remove any time or location information and check back into CVS. Using the OASIS-specific schema, http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd, will help with editing using a tool such as Oxygen. Copy the files into the _yyyy_/xml subdirectory. Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file; if you include other attributes they will overwrite what it in the database. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml wile with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. h2. B. Procedure for doing the actual production mooring turn So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. The steps to do this are outlined below. \[As of November 2007, there are several issues with making this a fool-proof set of instructions; therefore, for now they will serve as documentation for tasks  in the 2008 SSDS Hardening project.\] \\ # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id > 27060) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the instrument swap procedure and doing a deep delete on the wrong instrument deployment. If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: <251 elvis.shore.mbari.org /u/ssdsadmin> dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02 \- or wait for it to run with the hourly download on tsunami. #   #   #   Mike McCann (30 October 2007)]]> 6259091 3702785 3637251 2008-04-11 10:09:26.830 2008-04-11 10:09:26.830 2195618 2162852 11829376 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs and the path to the xml files to the newly deployed deviceIDs (aka ISI_IDs) and new xml directory. Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen or JEdit. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml file with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. \[Note: With the M1 - October 2009 deployment we added the individual inductive modem microcats as children of the mooring and also configured a parallel deployment of the TString that produced the same data. With the M2 - April 2010 deployment we configured the individual IM mirocrocats and did not configure a parallel TString deployment. This is a better model for the system and is much easier to configure and produces more easily consumed data by downstream processes such as combineTS.pl.\] With future deployments we will not configure TString. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /mbari/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. \\ For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed (or...as below \-rschramm 4/2010) \\ \\ {noformat} declare @myID as bigint set @myID = 27122 SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE id = @myID UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk in ( SELECT id FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID) ORDER BY id {noformat} h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSDS ingest will recognize them as new. # +Do not close (set endDates) the existing Test deployment until after the oasisToSSDS has run with the new platform deployment name+ and all new instrument deployments have been created in SSDS_Metadata. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadata, e.g.: {noformat} SELECT id, name, startDate, endDate, ParentID_FK FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') ORDER BY id DESC {noformat} Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file. # You will also need to edit the DStoNetCDF.pl script to set the new name of the M1 deployment, e.g.: {noformat} # # SSDS mooring name Deployment lookup # my %ssdsMooringDeplNames = ( M0 => 'M0', M1 => 'M1 - October 2009', M1Test => 'Test M1 - October 2009', M2 => 'M2 - April 2009', {noformat} # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. # When the mooring log message is sent to oasis with the actual time of deployment enter that as the startDate in the database.  All times (DTGs) in the SSDS_Metadata database are GMT. Make sure to set the startDate for the imctd microcat sensors too. This can be done with a query like this where you use the proper dataProduceIDs for the mooring and the imcd: {noformat} SELECT id, name, ParentID_FK, startDate, endDate, nominalDepth FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') AND (ParentID_FK = 31564) OR (ParentID_FK = 31548) ORDER BY id DESC {noformat} # Because of some bug in SSDS ingest the dataContainerTypes of the outputs from the instrument deployments do not get properly assigned the values of 'Stream'.  This needs to be fixed so that the NDBC datatransfers will work. To fix it edit the SSDS_Metadata database starting with a query like this: {noformat} SELECT ssdsdba.DataContainer.* FROM ssdsdba.DataProducer INNER JOIN ssdsdba.DataContainer ON ssdsdba.DataProducer.id = ssdsdba.DataContainer.DataProducerID_FK WHERE (ssdsdba.DataProducer.ParentID_FK = 31548) {noformat} where you use the DataProducerID for the new mooring deployment in the WHERE clause. ==> Change all of the 'File's in the dataContainerType' field to 'Stream's. Mike McCann (First edit: 30 October 2007, Last updated: 1 May 2009)]]> 11796617 11829375 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs and the path to the xml files to the newly deployed deviceIDs (aka ISI_IDs) and new xml directory. Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen or JEdit. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml file with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. \[Note: With the M1 - October 2009 deployment we added the individual inductive modem microcats as children of the mooring and also configured a parallel deployment of the TString that produced the same data. With the M2 - April 2010 deployment we configured the individual IM mirocrocats and did not configure a parallel TString deployment. This is a better model for the system and is much easier to configure and produces more easily consumed data by downstream processes such as combineTS.pl.\] With future deployments we will not configure TString. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /mbari/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. \\ For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed (or...as below \-rschramm 4/2010) \\ \\ {noformat} declare @myID as bigint set @myID = 27122 SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk in ( SELECT id FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID) ORDER BY id {noformat} h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSDS ingest will recognize them as new. # +Do not close (set endDates) the existing Test deployment until after the oasisToSSDS has run with the new platform deployment name+ and all new instrument deployments have been created in SSDS_Metadata. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadata, e.g.: {noformat} SELECT id, name, startDate, endDate, ParentID_FK FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') ORDER BY id DESC {noformat} Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file. # You will also need to edit the DStoNetCDF.pl script to set the new name of the M1 deployment, e.g.: {noformat} # # SSDS mooring name Deployment lookup # my %ssdsMooringDeplNames = ( M0 => 'M0', M1 => 'M1 - October 2009', M1Test => 'Test M1 - October 2009', M2 => 'M2 - April 2009', {noformat} # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. # When the mooring log message is sent to oasis with the actual time of deployment enter that as the startDate in the database.  All times (DTGs) in the SSDS_Metadata database are GMT. Make sure to set the startDate for the imctd microcat sensors too. This can be done with a query like this where you use the proper dataProduceIDs for the mooring and the imcd: {noformat} SELECT id, name, ParentID_FK, startDate, endDate, nominalDepth FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') AND (ParentID_FK = 31564) OR (ParentID_FK = 31548) ORDER BY id DESC {noformat} # Because of some bug in SSDS ingest the dataContainerTypes of the outputs from the instrument deployments do not get properly assigned the values of 'Stream'.  This needs to be fixed so that the NDBC datatransfers will work. To fix it edit the SSDS_Metadata database starting with a query like this: {noformat} SELECT ssdsdba.DataContainer.* FROM ssdsdba.DataProducer INNER JOIN ssdsdba.DataContainer ON ssdsdba.DataProducer.id = ssdsdba.DataContainer.DataProducerID_FK WHERE (ssdsdba.DataProducer.ParentID_FK = 31548) {noformat} where you use the DataProducerID for the new mooring deployment in the WHERE clause. ==> Change all of the 'File's in the dataContainerType' field to 'Stream's. Mike McCann (First edit: 30 October 2007, Last updated: 1 May 2009)]]> 11796616 11829373 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs and the path to the xml files to the newly deployed deviceIDs (aka ISI_IDs) and new xml directory. Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen or JEdit. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml file with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. \[Note: With the M1 - October 2009 deployment we added the individual inductive modem microcats as children of the mooring and also configured a parallel deployment of the TString that produced the same data. With the M2 - April 2010 deployment we configured the individual IM mirocrocats and did not configure a parallel TString deployment. This is a better model for the system and is much easier to configure and produces more easily consumed data by downstream processes such as combineTS.pl.\] With future deployments we will not configure TString. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /mbari/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed. h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSDS ingest will recognize them as new. # +Do not close (set endDates) the existing Test deployment until after the oasisToSSDS has run with the new platform deployment name+ and all new instrument deployments have been created in SSDS_Metadata. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadata, e.g.: {noformat} SELECT id, name, startDate, endDate, ParentID_FK FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') ORDER BY id DESC {noformat} Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file. # You will also need to edit the DStoNetCDF.pl script to set the new name of the M1 deployment, e.g.: {noformat} # # SSDS mooring name Deployment lookup # my %ssdsMooringDeplNames = ( M0 => 'M0', M1 => 'M1 - October 2009', M1Test => 'Test M1 - October 2009', M2 => 'M2 - April 2009', {noformat} # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. # When the mooring log message is sent to oasis with the actual time of deployment enter that as the startDate in the database.  All times (DTGs) in the SSDS_Metadata database are GMT. Make sure to set the startDate for the imctd microcat sensors too. This can be done with a query like this where you use the proper dataProduceIDs for the mooring and the imcd: {noformat} SELECT id, name, ParentID_FK, startDate, endDate, nominalDepth FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') AND (ParentID_FK = 31564) OR (ParentID_FK = 31548) ORDER BY id DESC {noformat} # Because of some bug in SSDS ingest the dataContainerTypes of the outputs from the instrument deployments do not get properly assigned the values of 'Stream'.  This needs to be fixed so that the NDBC datatransfers will work. To fix it edit the SSDS_Metadata database starting with a query like this: {noformat} SELECT ssdsdba.DataContainer.* FROM ssdsdba.DataProducer INNER JOIN ssdsdba.DataContainer ON ssdsdba.DataProducer.id = ssdsdba.DataContainer.DataProducerID_FK WHERE (ssdsdba.DataProducer.ParentID_FK = 31548) {noformat} where you use the DataProducerID for the new mooring deployment in the WHERE clause. ==> Change all of the 'File's in the dataContainerType' field to 'Stream's. Mike McCann (First edit: 30 October 2007, Last updated: 1 May 2009)]]> 11796614 2195610 2162844 19038487 19071253 1 2012-04-16 13:52:05.600 2012-04-16 13:52:05.600 19038486 2195609 tag. For instance any nominalLat/Lon/Depth. # If the tag has a 'name' attribute, make sure it does not have any deployment specific information in it. For example, 'ISUS Deployment' is better than 'M2 ISUS Deployment'. The reason for removing any deployment information from the XML is so that when the device moves to a different mooring, the user should not have to edit the XML. The goal is to get all the XML to a point where it never needs to be edited when an instrument is deployed (unless something in the way the data stream is generated from the instrument changes). # Go to the SSDS Device pages and verify that the all the device information (mfg, model, serial number, name, type, etc.) matches what is currently in SSDS. If any of those are different it will update the device information in SSDS when the XML comes in the data stream. # Verify RecordDescription and RecordVariables look correct. I usually go to the raw data pages in SSDS and bring up the last few packets from the device just to verify that the number of columns and bufferSeparator look about right. # Check any changes to the XML back into CVS. # Copy the XML to the \\Tornado\ssdsdata\mooring(m1\|m2)\YYYY\xml directory # Go to the \\Tornado\ssdsdata\mooring(m1\|m2)\YYYY\cfg directory. # This next step is the one that needs to be timed with the mooring turn. When the old instrument is shutdown: ## Open the ssds.cfg file in a text editor ## Find the line that shows the currently deployed instrument and copy it to a line just below it. For example, if we are replacing the GPS, it might look like this before: {panel:title=Before Copy} {noformat} instrument = PCO2,1471,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1471.xml instrument = Metsys,1480,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1480.xml instrument = GPS_TYPE3,1416,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1416.xml,,,TransformGPS instrument = Spec_PRR,1420,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1420.xml instrument = ADCP,1417,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1417.xml {noformat} {panel} and this after: {panel:title=After Copy} {noformat} instrument = PCO2,1471,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1471.xml instrument = Metsys,1480,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1480.xml instrument = GPS_TYPE3,1416,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1416.xml,,,TransformGPS instrument = GPS_TYPE3,1416,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1416.xml,,,TransformGPS instrument = Spec_PRR,1420,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1420.xml instrument = ADCP,1417,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1417.xml {noformat} {panel} # Now change the new line to have the correct device ID and the correct XML file URL {panel:title=After Device ID update} {noformat} instrument = PCO2,1471,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1471.xml instrument = Metsys,1480,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1480.xml instrument = GPS_TYPE3,1416,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1416.xml,,,TransformGPS instrument = GPS_TYPE3,1511,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1511.xml,,,TransformGPS instrument = Spec_PRR,1420,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1420.xml instrument = ADCP,1417,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1417.xml {noformat} {panel} # To clean up the previous deployment information, put start and end dates after the XML URL {panel:title=After Adding Start/End dates} {noformat} instrument = PCO2,1471,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1471.xml instrument = Metsys,1480,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1480.xml instrument = GPS_TYPE3,1416,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1416.xml,2007/04/25 16:21:58,2007/08/01 10:00:00,TransformGPS instrument = GPS_TYPE3,1511,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1511.xml,,,TransformGPS instrument = Spec_PRR,1420,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1420.xml instrument = ADCP,1417,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1417.xml {noformat} {panel} # Save the cfg file. {note:title=Saving the file will make the change take hold}When the ssds.cfg file changes (saved) is when the OASIS2SSDS processing will pick up the instrument change. Now, the next time it runs it will pick up the instrument change, grab the XML file from the 'xml' directory and publish it to SSDS. It will then publish all data under the new device ID. {note} # Edit the metadata in SSDS to put a close date on the old instrument deployment in SSDS. Currently I do that using Enterprise Manager. h5. If this is being done after the fact the next steps will also need to be taken. # After the new deployment shows up in SSDS, the start time for the new deployment will need to be edited to match the actual time the instrument was swapped. # Also because the data was being published under the incorrect device ID, it will need to be moved from one database table in SSDS_Data on Solstice to another table. ## The first thing that I do is grab the timestamp from the last packet sent from the old device in the raw data page on SSDS. ## For example, I go to: [http://new-ssds.mbari.org:8080/ssds/siamRawDataStep1.jsp]and enter the old device ID and set the number of packets back to make sure it goes far enough back to cover the actual time of the instrument swap. Then click on 'Next->'. ## Once the raw data shows up, find the last packet from the old device and grab the 'SIAM Timestamp' value (not the date/time) as that will be used in the Enterprise Manager query. ## Open Enterprise Manager and navigate to the 'SSDS_Data' database on Solstice. ## Browse the tables and find the table with the device ID of the old device and right click on it and select 'Open Table->Return all rows'. ## Click on the 'SQL' button in Enterprise Manager to bring up the SQL pane. It should show the basic query which should look something like this: {noformat} SELECT * FROM [1416] {noformat} ## Now add the where clause to pick only the data that is after the timestamp you grabbed from the last packet on the web page. {note:title=Timestamps in SQL are in Seconds}A quick note here, the 'SIAM Timestamp' on the raw data page is actually in milliseconds and the database column is in seconds so you will have to remove the last three digits of the 'SIAM Timestamp' before putting it in this query. {note} {noformat} SELECT * FROM [1416] WHERE timestampSeconds > 1185963023 {noformat} ## Run this query by clicking the run button '\!' in Enterprise Manager. ## Look over the results to make sure they look about right (usually you are looking for the length of the return which should be much shorter). You can actually use a count query to see how many rows this query will return. A count query would look like: {noformat} SELECT count(*) FROM [1416] WHERE timestampSeconds > 1185963023 {noformat} ## Once you know the query is correct, copy it to the clipboard and close the query window in Enterprise Manager. ## Navigate to the table of the device you want to copy the data into and right click and select 'All Tasks->Import Data...' which will fire up the DTS wizard. ### Click on 'Next>' ### For the Data Source database choose Solstice ### Select the 'SSDS_Data' database (note you should have permissions to do all this and use your windows authentication) ### Click on 'Next>' ### The destination configuration should be already to go (Solistice and SSDS_Data database). ### Click on 'Next>' ### Select 'Use a query to specify the data to transfer' ### Click on 'Next>' ### Paste the query from your clipboard into the 'Query Statement' window (You can click on 'Parse' if you want a quick sanity check) ### Click on 'Next>' ### Click on the 'Results' entry under the 'Destination' column which will enable a drop down box. ### Choose the table of the newly installed device where you will be copying the data to. ### Click on 'Next>' ### Click on 'Next>' ### Click on 'Finish' which will copy the data. ### Once that is done, open the table of the old instrument and the SQL pane so that we can construct the delete query on the old data. ### Paste in the select query and verify it is the same data you copied over: {noformat} SELECT * FROM [1416] WHERE timestampSeconds > 1185963023 {noformat} ### If it looks good, click on the 'Change Query type ...' button in Enterprise Manager and select 'Delete'. This will change the query to a delete query. ### Run the query by click on the run '\!' button. That will remove all the data from the old instrument. That's it ... whew\! Kevin Gomes (August 3, 2007)]]> 2162843 2195608 tag. For instance any nominalLat/Lon/Depth. # If the tag has a 'name' attribute, make sure it does not have any deployment specific information in it. For example, 'ISUS Deployment' is better than 'M2 ISUS Deployment'. The reason for removing any deployment information from the XML is so that when the device moves to a different mooring, the user should not have to edit the XML. The goal is to get all the XML to a point where it never needs to be edited when an instrument is deployed (unless something in the way the data stream is generated from the instrument changes). # Go to the SSDS Device pages and verify that the all the device information (mfg, model, serial number, name, type, etc.) matches what is currently in SSDS. If any of those are different it will update the device information in SSDS when the XML comes in the data stream. # Verify RecordDescription and RecordVariables look correct. I usually go to the raw data pages in SSDS and bring up the last few packets from the device just to verify that the number of columns and bufferSeparator look about right. # Check any changes to the XML back into CVS. # Copy the XML to the \\Tornado\ssdsdata\mooring(m1\|m2)\YYYY\xml directory # Go to the \\Tornado\ssdsdata\mooring(m1\|m2)\YYYY\cfg directory. # This next step is the one that needs to be timed with the mooring turn. When the old instrument is shutdown: ## Open the ssds.cfg file in a text editor ## Find the line that shows the currently deployed instrument and copy it to a line just below it. For example, if we are replacing the GPS, it might look like this before: {panel:title=Before Copy} {noformat} instrument = PCO2,1471,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1471.xml instrument = Metsys,1480,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1480.xml instrument = GPS_TYPE3,1416,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1416.xml,,,TransformGPS instrument = Spec_PRR,1420,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1420.xml instrument = ADCP,1417,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1417.xml {noformat} {panel} and this after: {panel:title=After Copy} {noformat} instrument = PCO2,1471,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1471.xml instrument = Metsys,1480,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1480.xml instrument = GPS_TYPE3,1416,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1416.xml,,,TransformGPS instrument = GPS_TYPE3,1416,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1416.xml,,,TransformGPS instrument = Spec_PRR,1420,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1420.xml instrument = ADCP,1417,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1417.xml {noformat} {panel} # Now change the new line to have the correct device ID and the correct XML file URL {panel:title=After Device ID update} {noformat} instrument = PCO2,1471,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1471.xml instrument = Metsys,1480,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1480.xml instrument = GPS_TYPE3,1416,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1416.xml,,,TransformGPS instrument = GPS_TYPE3,1511,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1511.xml,,,TransformGPS instrument = Spec_PRR,1420,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1420.xml instrument = ADCP,1417,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1417.xml {noformat} {panel} # To clean up the previous deployment information, put start and end dates after the XML URL {panel:title=After Adding Start/End dates} {noformat} instrument = PCO2,1471,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1471.xml instrument = Metsys,1480,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1480.xml instrument = GPS_TYPE3,1416,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1416.xml,2007/04/25 16:21:58,2007/08/01 10:00:00,TransformGPS instrument = GPS_TYPE3,1511,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1511.xml,,,TransformGPS instrument = Spec_PRR,1420,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1420.xml instrument = ADCP,1417,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1417.xml {noformat} {panel} # Save the cfg file. {note:title=Saving the file will make the change take hold}When the ssds.cfg file changes (saved) is when the OASIS2SSDS processing will pick up the instrument change. Now, the next time it runs it will pick up the instrument change, grab the XML file from the 'xml' directory and publish it to SSDS. It will then publish all data under the new device ID. {note} # Edit the metadata in SSDS to put a close date on the old instrument deployment in SSDS. Currently I do that using Enterprise Manager. h5. If this is being done after the fact the next steps will also need to be taken. # After the new deployment shows up in SSDS, the start time for the new deployment will need to be edited to match the actual time the instrument was swapped. # Also because the data was being published under the incorrect device ID, it will need to be moved from one database table in SSDS_Data on Solstice to another table. ## The first thing that I do is grab the timestamp from the last packet sent from the old device in the raw data page on SSDS. ## For example, I go to: [http://new-ssds.mbari.org:8080/ssds/siamRawDataStep1.jsp]and enter the old device ID and set the number of packets back to make sure it goes far enough back to cover the actual time of the instrument swap. Then click on 'Next->'. ## Once the raw data shows up, find the last packet from the old device and grab the 'SIAM Timestamp' value (not the date/time) as that will be used in the Enterprise Manager query. ## Open Enterprise Manager and navigate to the 'SSDS_Data' database on Solstice. ## Browse the tables and find the table with the device ID of the old device and right click on it and select 'Open Table->Return all rows'. ## Click on the 'SQL' button in Enterprise Manager to bring up the SQL pane. It should show the basic query which should look something like this: {noformat} SELECT * FROM [1416] {noformat} ## Now add the where clause to pick only the data that is after the timestamp you grabbed from the last packet on the web page. {note:title=Timestamps in SQL are in Seconds}A quick note here, the 'SIAM Timestamp' on the raw data page is actually in milliseconds and the database column is in seconds so you will have to remove the last three digits of the 'SIAM Timestamp' before putting it in this query. {note} {noformat} SELECT * FROM [1416] WHERE timestampSeconds > 1185963023 {noformat} ## Run this query by clicking the run button '\!' in Enterprise Manager. ## Look over the results to make sure they look about right (usually you are looking for the length of the return which should be much shorter). You can actually use a count query to see how many rows this query will return. A count query would look like: {noformat} SELECT count(*) FROM [1416] WHERE timestampSeconds > 1185963023 {noformat} ## Once you know the query is correct, copy it to the clipboard and close the query window in Enterprise Manager. ## Navigate to the table of the device you want to copy the data into and right click and select 'All Tasks->Import Data...' which will fire up the DTS wizard. ### Click on 'Next>' ### For the Data Source database choose Solstice ### Select the 'SSDS_Data' database (note you should have permissions to do all this and use your windows authentication) ### Click on 'Next>' ### The destination configuration should be already to go (Solistice and SSDS_Data database). ### Click on 'Next>' ### Select 'Use a query to specify the data to transfer' ### Click on 'Next>' ### Paste the query from your clipboard into the 'Query Statement' window (You can click on 'Parse' if you want a quick sanity check) ### Click on 'Next>' ### Click on the 'Results' entry under the 'Destination' column which will enable a drop down box. ### Choose the table of the newly installed device where you will be copying the data to. ### Click on 'Next>' ### Click on 'Next>' ### Click on 'Finish' which will copy the data. ### Once that is done, open the table of the old instrument and the SQL pane so that we can construct the delete query on the old data. ### Paste in the select query and verify it is the same data you copied over: {noformat} SELECT * FROM [1416] WHERE timestampSeconds > 1185963023 {noformat} ### If it looks good, click on the 'Change Query type ...' button in Enterprise Manager and select 'Delete'. This will change the query to a delete query. ### Run the query by click on the run '\!' button. That will remove all the data from the old instrument. That's it ... whew\! Kevin Gomes (August 3, 2007)]]> 2162842 2195606 tag. For instance any nominalLat/Lon/Depth. # If the tag has a 'name' attribute, make sure it does not have any deployment specific information in it. For example, 'ISUS Deployment' is better than 'M2 ISUS Deployment'. The reason for removing any deployment information from the XML is so that when the device moves to a different mooring, the user should not have to edit the XML. The goal is to get all the XML to a point where it never needs to be edited when an instrument is deployed (unless something in the way the data stream is generated from the instrument changes). # Go to the SSDS Device pages and verify that the all the device information (mfg, model, serial number, name, type, etc.) matches what is currently in SSDS. If any of those are different it will update the device information in SSDS when the XML comes in the data stream. # Verify RecordDescription and RecordVariables look correct. I usually go to the raw data pages in SSDS and bring up the last few packets from the device just to verify that the number of columns and bufferSeparator look about right. # Check any changes to the XML back into CVS. # Copy the XML to the \\Tornado\ssdsdata\mooring(m1\|m2)\YYYY\xml directory # Go to the \\Tornado\ssdsdata\mooring(m1\|m2)\YYYY\cfg directory. # This next step is the one that needs to be timed with the mooring turn. When the old instrument is shutdown: ## Open the ssds.cfg file in a text editor ## Find the line that shows the currently deployed instrument and copy it to a line just below it. For example, if we are replacing the GPS, it might look like this before: {panel:title=Before Copy} {noformat} instrument = PCO2,1471,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1471.xml instrument = Metsys,1480,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1480.xml instrument = GPS_TYPE3,1416,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1416.xml,,,TransformGPS instrument = Spec_PRR,1420,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1420.xml instrument = ADCP,1417,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1417.xml {noformat} {panel} and this after: {panel:title=After Copy} {noformat} instrument = PCO2,1471,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1471.xml instrument = Metsys,1480,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1480.xml instrument = GPS_TYPE3,1416,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1416.xml,,,TransformGPS instrument = GPS_TYPE3,1416,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1416.xml,,,TransformGPS instrument = Spec_PRR,1420,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1420.xml instrument = ADCP,1417,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1417.xml {noformat} {panel} # Now change the new line to have the correct device ID and the correct XML file URL {panel:title=After Device ID update} {noformat} instrument = PCO2,1471,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1471.xml instrument = Metsys,1480,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1480.xml instrument = GPS_TYPE3,1416,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1416.xml,,,TransformGPS instrument = GPS_TYPE3,1511,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1511.xml,,,TransformGPS instrument = Spec_PRR,1420,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1420.xml instrument = ADCP,1417,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1417.xml {noformat} {panel} # To clean up the previous deployment information, put start and end dates after the XML URL {panel:title=After Adding Start/End dates} {noformat} instrument = PCO2,1471,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1471.xml instrument = Metsys,1480,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1480.xml instrument = GPS_TYPE3,1416,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1416.xml,2007/04/25 16:21:58,2007/08/01 10:00:00,TransformGPS instrument = GPS_TYPE3,1511,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1511.xml,,,TransformGPS instrument = Spec_PRR,1420,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1420.xml instrument = ADCP,1417,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1417.xml {noformat} {panel} # Save the cfg file. {note:title=Saving the file will make the change take hold}When the ssds.cfg file changes (saved) is when the OASIS2SSDS processing will pick up the instrument change. Now, the next time it runs it will pick up the instrument change, grab the XML file from the 'xml' directory and publish it to SSDS. It will then publish all data under the new device ID. {note} # Edit the metadata in SSDS to put a close date on the old instrument deployment in SSDS. Currently I do that using Enterprise Manager. h5. If this is being done after the fact the next steps will also need to be taken. # After the new deployment shows up in SSDS, the start time for the new deployment will need to be edited to match the actual time the instrument was swapped. # Also because the data was being published under the incorrect device ID, it will need to be moved from one database table in SSDS_Data on Solstice to another table. ## The first thing that I do is grab the timestamp from the last packet sent from the old device in the raw data page on SSDS. ## For example, I go to: [http://new-ssds.mbari.org:8080/ssds/siamRawDataStep1.jsp]and enter the old device ID and set the number of packets back to make sure it goes far enough back to cover the actual time of the instrument swap. Then click on 'Next->'. ## Once the raw data shows up, find the last packet from the old device and grab the 'SIAM Timestamp' value (not the date/time) as that will be used in the Enterprise Manager query. ## Open Enterprise Manager and navigate to the 'SSDS_Data' database on Solstice. ## Browse the tables and find the table with the device ID of the old device and right click on it and select 'Open Table->Return all rows'. ## Click on the 'SQL' button in Enterprise Manager to bring up the SQL pane. It should show the basic query which should look something like this: {noformat} SELECT * FROM [1416] {noformat} ## Now add the where clause to pick only the data that is after the timestamp you grabbed from the last packet on the web page. {note:title=Timestamps in SQL are in Seconds}A quick note here, the 'SIAM Timestamp' on the raw data page is actually in milliseconds and the database column is in seconds so you will have to remove the last three digits of the 'SIAM Timestamp' before putting it in this query. {note} {noformat} SELECT * FROM [1416] WHERE timestampSeconds > 1185963023 {noformat} ## Run this query by clicking the run button '\!' in Enterprise Manager. ## Look over the results to make sure they look about right (usually you are looking for the length of the return which should be much shorter). You can actually use a count query to see how many rows this query will return. A count query would look like: {noformat} SELECT count(*) FROM [1416] WHERE timestampSeconds > 1185963023 {noformat} ## Once you know the query is correct, copy it to the clipboard and close the query window in Enterprise Manager. ## Navigate to the table of the device you want to copy the data into and right click and select 'All Tasks->Import Data...' which will fire up the DTS wizard. ### Click on 'Next>' ### For the Data Source database choose Solstice ### Select the 'SSDS_Data' database (note you should have permissions to do all this and use your windows authentication) ### Click on 'Next>' ### The destination configuration should be already to go (Solistice and SSDS_Data database). ### Click on 'Next>' ### Select 'Use a query to specify the data to transfer' ### Click on 'Next>' ### Paste the query from your clipboard into the 'Query Statement' window (You can click on 'Parse' if you want a quick sanity check) ### Click on 'Next>' ### Click on the 'Results' entry under the 'Destination' column which will enable a drop down box. ### Choose the table of the newly installed device where you will be copying the data to. ### Click on 'Next>' ### Click on 'Next>' ### Click on 'Finish' which will copy the data. ### Once that is done, open the table of the old instrument and the SQL pane so that we can construct the delete query on the old data. ### Paste in the select query and verify it is the same data you copied over: {noformat} SELECT * FROM [1416] WHERE timestampSeconds > 1185963023 {noformat} ### If it looks good, click on the 'Change Query type ...' button in Enterprise Manager and select 'Delete'. This will change the query to a delete query. ### Run the query by click on the run '\!' button. That will remove all the data from the old instrument. That's it ... whew\! Kevin Gomes (August 3, 2007)]]> 2162840 9404897 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs to the newly deployed deviceIDs (aka ISI_IDs). Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what it in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml wile with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /hosts/tornado_vol0/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed. h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSD ingest will recognize them as new. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /hosts/tornado_vol0/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query like this where the DataProducer.id is >= the id of the new platform, e.g.: {noformat} {noformat} # ;sd;s #   #   # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. #   # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DPLOYMENTS file. # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. Mike McCann (First edit: 30 October 2007, Last updated: 8 October 2008)]]> 9372156 2195605 tag. For instance any nominalLat/Lon/Depth. # If the tag has a 'name' attribute, make sure it does not have any deployment specific information in it. For example, 'ISUS Deployment' is better than 'M2 ISUS Deployment'. The reason for removing any deployment information from the XML is so that when the device moves to a different mooring, the user should not have to edit the XML. The goal is to get all the XML to a point where it never needs to be edited when an instrument is deployed (unless something in the way the data stream is generated from the instrument changes). # Go to the SSDS Device pages and verify that the all the device information (mfg, model, serial number, name, type, etc.) matches what is currently in SSDS. If any of those are different it will update the device information in SSDS when the XML comes in the data stream. # Verify RecordDescription and RecordVariables look correct. I usually go to the raw data pages in SSDS and bring up the last few packets from the device just to verify that the number of columns and bufferSeparator look about right. # Check any changes to the XML back into CVS. # Copy the XML to the \\Tornado\ssdsdata\mooring(m1\|m2)\YYYY\xml directory # Go to the \\Tornado\ssdsdata\mooring(m1\|m2)\YYYY\cfg directory. # This next step is the one that needs to be timed with the mooring turn. When the old instrument is shutdown: ## Open the ssds.cfg file in a text editor ## Find the line that shows the currently deployed instrument and copy it to a line just below it. For example, if we are replacing the GPS, it might look like this before: {panel:title=Before Copy} {noformat} instrument = PCO2,1471,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1471.xml instrument = Metsys,1480,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1480.xml instrument = GPS_TYPE3,1416,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1416.xml,,,TransformGPS instrument = Spec_PRR,1420,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1420.xml instrument = ADCP,1417,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1417.xml {noformat} {panel} and this after: {panel:title=After Copy} {noformat} instrument = PCO2,1471,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1471.xml instrument = Metsys,1480,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1480.xml instrument = GPS_TYPE3,1416,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1416.xml,,,TransformGPS instrument = GPS_TYPE3,1416,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1416.xml,,,TransformGPS instrument = Spec_PRR,1420,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1420.xml instrument = ADCP,1417,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1417.xml {noformat} {panel} # Now change the new line to have the correct device ID and the correct XML file URL {panel:title=After Device ID update} {noformat} instrument = PCO2,1471,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1471.xml instrument = Metsys,1480,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1480.xml instrument = GPS_TYPE3,1416,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1416.xml,,,TransformGPS instrument = GPS_TYPE3,1511,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1511.xml,,,TransformGPS instrument = Spec_PRR,1420,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1420.xml instrument = ADCP,1417,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1417.xml {noformat} {panel} # To clean up the previous deployment information, put start and end dates after the XML URL {panel:title=After Adding Start/End dates} {noformat} instrument = PCO2,1471,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1471.xml instrument = Metsys,1480,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1480.xml instrument = GPS_TYPE3,1416,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1416.xml,2007/04/25 16:21:58,2007/08/01 10:00:00,TransformGPS instrument = GPS_TYPE3,1511,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1511.xml,,,TransformGPS instrument = Spec_PRR,1420,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1420.xml instrument = ADCP,1417,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1417.xml {noformat} {panel} # Save the cfg file. {note:title=Saving the file will make the change take hold}When the ssds.cfg file changes (saved) is when the OASIS2SSDS processing will pick up the instrument change. Now, the next time it runs it will pick up the instrument change, grab the XML file from the 'xml' directory and publish it to SSDS. It will then publish all data under the new device ID. {note} # Edit the metadata in SSDS to put a close date on the old instrument deployment in SSDS. Currently I do that using Enterprise Manager. h5. If this is being done after the fact the next steps will also need to be taken. # After the new deployment shows up in SSDS, the start time for the new deployment will need to be edited to match the actual time the instrument was swapped. # Also because the data was being published under the incorrect device ID, it will need to be moved from one database table in SSDS_Data on Solstice to another table. ## The first thing that I do is grab the timestamp from the last packet sent from the old device in the raw data page on SSDS. ## For example, I go to: [http://new-ssds.mbari.org:8080/ssds/siamRawDataStep1.jsp]and enter the old device ID and set the number of packets back to make sure it goes far enough back to cover the actual time of the instrument swap. Then click on 'Next->'. ## Once the raw data shows up, find the last packet from the old device and grab the 'SIAM Timestamp' value (not the date/time) as that will be used in the Enterprise Manager query. ## Open Enterprise Manager and navigate to the 'SSDS_Data' database on Solstice. ## Browse the tables and find the table with the device ID of the old device and right click on it and select 'Open Table->Return all rows'. ## Click on the 'SQL' button in Enterprise Manager to bring up the SQL pane. It should show the basic query which should look something like this: {noformat} SELECT * FROM [1416] {noformat} ## Now add the where clause to pick only the data that is after the timestamp you grabbed from the last packet on the web page. {note:title=Timestamps in SQL are in Seconds}A quick note here, the 'SIAM Timestamp' on the raw data page is actually in milliseconds and the database column is in seconds so you will have to remove the last three digits of the 'SIAM Timestamp' before putting it in this query. {note} {noformat} SELECT * FROM [1416] WHERE timestampSeconds > 1185963023 {noformat} ## Run this query by clicking the run button '\!' in Enterprise Manager. ## Look over the results to make sure they look about right (usually you are looking for the length of the return which should be much shorter). You can actually use a count query to see how many rows this query will return. A count query would look like: {noformat} SELECT count(*) FROM [1416] WHERE timestampSeconds > 1185963023 {noformat} ## Once you know the query is correct, copy it to the clipboard and close the query window in Enterprise Manager. ## Navigate to the table of the device you want to copy the data into and right click and select 'All Tasks->Import Data...' which will fire up the DTS wizard. ### Click on 'Next>' ### For the Data Source database choose Solstice ### Select the 'SSDS_Data' database (note you should have permissions to do all this and use your windows authentication) ### Click on 'Next>' ### The destination configuration should be already to go (Solistice and SSDS_Data database). ### Click on 'Next>' ### Select 'Use a query to specify the data to transfer' ### Click on 'Next>' ### Paste the query from your clipboard into the 'Query Statement' window (You can click on 'Parse' if you want a quick sanity check) ### Click on 'Next>' ### Click on the 'Results' entry under the 'Destination' column which will enable a drop down box. ### Choose the table of the newly installed device where you will be copying the data to. ### Click on 'Next>' ### Click on 'Next>' ### Click on 'Finish' which will copy the data. ### Once that is done, open the table of the old instrument and the SQL pane so that we can construct the delete query on the old data. ### Paste in the select query and verify it is the same data you copied over: {noformat} SELECT * FROM [1416] WHERE timestampSeconds > 1185963023 {noformat} ### If it looks good, click on the 'Change Query type ...' button in Enterprise Manager and select 'Delete'. This will change the query to a delete query. ### Run the query by click on the run '\!' button. That will remove all the data from the old instrument. That's it ... whew\! Kevin Gomes (August 3, 2007)]]> 2162839 2195604 tag. For instance any nominalLat/Lon/Depth. # If the tag has a 'name' attribute, make sure it does not have any deployment specific information in it. For example, 'ISUS Deployment' is better than 'M2 ISUS Deployment'. The reason for removing any deployment information from the XML is so that when the device moves to a different mooring, the user should not have to edit the XML. The goal is to get all the XML to a point where it never needs to be edited when an instrument is deployed (unless something in the way the data stream is generated from the instrument changes). # Go to the SSDS Device pages and verify that the all the device information (mfg, model, serial number, name, type, etc.) matches what is currently in SSDS. If any of those are different it will update the device information in SSDS when the XML comes in the data stream. # Verify RecordDescription and RecordVariables look correct. I usually go to the raw data pages in SSDS and bring up the last few packets from the device just to verify that the number of columns and bufferSeparator look about right. # Check any changes to the XML back into CVS. # Copy the XML to the \\Tornado\ssdsdata\mooring(m1\|m2)\YYYY\xml directory # Go to the \\Tornado\ssdsdata\mooring(m1\|m2)\YYYY\cfg directory. # This next step is the one that needs to be timed with the mooring turn. When the old instrument is shutdown: ## Open the ssds.cfg file in a text editor ## Find the line that shows the currently deployed instrument and copy it to a line just below it. For example, if we are replacing the GPS, it might look like this before: {panel:title=Before Copy} {noformat} instrument = PCO2,1471,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1471.xml instrument = Metsys,1480,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1480.xml instrument = GPS_TYPE3,1416,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1416.xml,,,TransformGPS instrument = Spec_PRR,1420,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1420.xml instrument = ADCP,1417,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1417.xml {noformat} {panel} and this after: {panel:title=After Copy} {noformat} instrument = PCO2,1471,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1471.xml instrument = Metsys,1480,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1480.xml instrument = GPS_TYPE3,1416,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1416.xml,,,TransformGPS instrument = GPS_TYPE3,1416,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1416.xml,,,TransformGPS instrument = Spec_PRR,1420,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1420.xml instrument = ADCP,1417,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1417.xml {noformat} {panel} # Now change the new line to have the correct device ID and the correct XML file URL {panel:title=After Device ID update} {noformat} instrument = PCO2,1471,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1471.xml instrument = Metsys,1480,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1480.xml instrument = GPS_TYPE3,1416,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1416.xml,,,TransformGPS instrument = GPS_TYPE3,1511,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1511.xml,,,TransformGPS instrument = Spec_PRR,1420,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1420.xml instrument = ADCP,1417,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1417.xml {noformat} {panel} # To clean up the previous deployment information, put start and end dates after the XML URL {panel:title=After Adding Start/End dates} {noformat} instrument = PCO2,1471,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1471.xml instrument = Metsys,1480,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1480.xml instrument = GPS_TYPE3,1416,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1416.xml,2007/04/25 16:21:58,2007/08/01 10:00:00,TransformGPS instrument = GPS_TYPE3,1511,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1511.xml,,,TransformGPS instrument = Spec_PRR,1420,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1420.xml instrument = ADCP,1417,http://dods.mbari.org/data/ssdsdata/mooring/m2/2007/xml/1417.xml {noformat} {panel} # Save the cfg file. {note:title=Saving the file will make the change take hold}When the ssds.cfg file changes (saved) is when the OASIS2SSDS processing will pick up the instrument change. Now, the next time it runs it will pick up the instrument change, grab the XML file from the 'xml' directory and publish it to SSDS. It will then publish all data under the new device ID. {note} # Edit the metadata in SSDS to put a close date on the old instrument deployment in SSDS. Currently I do that using Enterprise Manager. h5. If this is being done after the fact the next steps will also need to be taken. # After the new deployment shows up in SSDS, the start time for the new deployment will need to be edited to match the actual time the instrument was swapped. # Also because the data was being published under the incorrect device ID, it will need to be moved from one database table in SSDS_Data on Solstice to another table. ## The first thing that I do is grab the timestamp from the last packet sent from the old device in the raw data page on SSDS. ## For example, I go to: [http://new-ssds.mbari.org:8080/ssds/siamRawDataStep1.jsp]and enter the old device ID and set the number of packets back to make sure it goes far enough back to cover the actual time of the instrument swap. Then click on 'Next->'. ## Once the raw data shows up, find the last packet from the old device and grab the 'SIAM Timestamp' value (not the date/time) as that will be used in the Enterprise Manager query. ## Open Enterprise Manager and navigate to the 'SSDS_Data' database on Solstice. ## Browse the tables and find the table with the device ID of the old device and right click on it and select 'Open Table->Return all rows'. ## Click on the 'SQL' button in Enterprise Manager to bring up the SQL pane. It should show the basic query which should look something like this: {noformat} SELECT * FROM [1416] {noformat} ## Now add the where clause to pick only the data that is after the timestamp you grabbed from the last packet on the web page. {note:title=Timestamps in SQL are in Seconds}A quick note here, the 'SIAM Timestamp' on the raw data page is actually in milliseconds and the database column is in seconds so you will have to remove the last three digits of the 'SIAM Timestamp' before putting it in this query. {note} {noformat} SELECT * FROM [1416] WHERE timestampSeconds > 1185963023 {noformat} ## Run this query by clicking the run button '\!' in Enterprise Manager. ## Look over the results to make sure they look about right (usually you are looking for the length of the return which should be much shorter). You can actually use a count query to see how many rows this query will return. A count query would look like: {noformat} SELECT count(*) FROM [1416] WHERE timestampSeconds > 1185963023 {noformat} ## Once you know the query is correct, copy it to the clipboard and close the query window in Enterprise Manager. ## Navigate to the table of the device you want to copy the data into and right click and select 'All Tasks->Import Data...' which will fire up the DTS wizard. ### Click on 'Next>' ### For the Data Source database choose Solstice ### Select the 'SSDS_Data' database (note you should have permissions to do all this and use your windows authentication) ### Click on 'Next>' ### The destination configuration should be already to go (Solistice and SSDS_Data database). ### Click on 'Next>' ### Select 'Use a query to specify the data to transfer' ### Click on 'Next>' ### Paste the query from your clipboard into the 'Query Statement' window (You can click on 'Parse' if you want a quick sanity check) ### Click on 'Next>' ### Click on the 'Results' entry under the 'Destination' column which will enable a drop down box. ### Choose the table of the newly installed device where you will be copying the data to. ### Click on 'Next>' ### Click on 'Next>' ### Click on 'Finish' which will copy the data. ### Once that is done, open the table of the old instrument and the SQL pane so that we can construct the delete query on the old data. ### Paste in the select query and verify it is the same data you copied over: {noformat} SELECT * FROM [1416] WHERE timestampSeconds > 1185963023 {noformat} ### If it looks good, click on the 'Change Query type ...' button in Enterprise Manager and select 'Delete'. This will change the query to a delete query. ### Run the query by click on the run '\!' button. That will remove all the data from the old instrument. That's it ... whew\! Kevin Gomes (August 3, 2007)]]> 2162838 2195603 2162837 9404896 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs to the newly deployed deviceIDs (aka ISI_IDs). Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen. Copy the files into the _yyyy_/xml subdirectory. Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file; if you include other attributes they will overwrite what it in the database. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml wile with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /hosts/tornado_vol0/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed. h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSD ingest will recognize them as new. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /hosts/tornado_vol0/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query like this where the DataProducer.id is >= the id of the new platform, e.g.: {noformat} {noformat} # ;sd;s #   #   # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. #   # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DPLOYMENTS file. # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. Mike McCann (First edit: 30 October 2007, Last updated: 8 October 2008)]]> 9372155 9404870 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. # Edit the mooring .cfg file and changed the instrument deviceIDs to the newly deployed deviceIDs (aka ISI_IDs). Refer to email from OSG for the correct numbers, e.g.: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen. Copy the files into the _yyyy_/xml subdirectory. Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file; if you include other attributes they will overwrite what it in the database. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml wile with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /hosts/tornado_vol0/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed. h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSD ingest will recognize them as new. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /hosts/tornado_vol0/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query like this where the DataProducer.id is >= the id of the new platform, e.g.: {noformat} {noformat} # ;sd;s #   #   # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. #   # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DPLOYMENTS file. # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. Mike McCann (First edit: 30 October 2007, Last updated: 8 October 2008)]]> 9372126 7798893 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. # Edit the mooring .cfg file and changed the instrument deviceIDs to the newly deployed deviceIDs (aka ISI_IDs). Refer to email from OSG for the correct numbers, e.g.: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application, e.g.: {noformat} {noformat} Remove any time or location information and check back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen. Copy the files into the _yyyy_/xml subdirectory. Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file; if you include other attributes they will overwrite what it in the database. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml wile with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /hosts/tornado_vol0/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed. h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSD ingest will recognize them as new. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /hosts/tornado_vol0/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query like this where the DataProducer.id is >= the id of the new platform, e.g.: {noformat} {noformat} # ;sd;s #   #   # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. #   # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DPLOYMENTS file. # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. Mike McCann (First edit: 30 October 2007, Last updated: 8 October 2008)]]> 7766129 3670019 3637251 2195637 27060) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: <251 elvis.shore.mbari.org /u/ssdsadmin> dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02 \- or wait for it to run with the hourly download on tsunami. #   #   #   Mike McCann (30 October 2007)]]> 2162871 2195631 tm_year + 1900)) + tp->tm_yday + 1; else itime = (1000 * tp->tm_year) + tp->tm_yday + 1; dtime = tp->tm_yday + 1.0 + (double)((3600 * tp->tm_hour) + (60 * tp->tm_min) + tp->tm_sec) / 86400.0; len_got = cc - 9; /* compute amount of log data gotten*/ if ( len_got > 0 ) memmove( buffer, buffer + 9, sizeof(buffer) - 9 ); else if ( len_got < 0 ) /* Move log data to start of buffer*/ { printf("Incomplete record header in %s, record %d (type %d)\n",filename, hdr.log_nmbr,hdr.log_type); continue; } : : : : ]]> 2162865 2195632 tm_year + 1900)) + tp->tm_yday + 1; else itime = (1000 * tp->tm_year) + tp->tm_yday + 1;dtime = tp->tm_yday + 1.0 + (double)((3600 * tp->tm_hour) + (60 * tp->tm_min) + tp->tm_sec) / 86400.0;len_got = cc - 9; /* compute amount of log data gotten*/ if ( len_got > 0 ) memmove( buffer, buffer + 9, sizeof(buffer) - 9 ); else if ( len_got < 0 ) /* Move log data to start of buffer*/{ printf("Incomplete record header in %s, record %d (type %d)\n",filename, hdr.log_nmbr,hdr.log_type);continue; }: : : : {noformat} {noformat} {noformat}]]> 2162866 2195633 tm_year + 1900)) + tp->tm_yday + 1; else itime = (1000 * tp->tm_year) + tp->tm_yday + 1;dtime = tp->tm_yday + 1.0 + (double)((3600 * tp->tm_hour) + (60 * tp->tm_min) + tp->tm_sec) / 86400.0;len_got = cc - 9; /* compute amount of log data gotten*/ if ( len_got > 0 ) memmove( buffer, buffer + 9, sizeof(buffer) - 9 ); else if ( len_got < 0 ) /* Move log data to start of buffer*/{ printf("Incomplete record header in %s, record %d (type %d)\n",filename, hdr.log_nmbr,hdr.log_type);continue; }: : : : {noformat} {noformat} {noformat}]]> 2162867 2195627 tm_year + 1900)) + tp->tm_yday + 1;**            else{*}*                itime = (1000 * tp->tm_year) + tp->tm_yday + 1;* *            dtime = tp->tm_yday + 1.0 + (double)((3600 * tp->tm_hour)**                                    + (60 * tp->tm_min) + tp->tm_sec) / 86400.0;* *            len_got = cc - 9;                        /\* compute amount of log data gotten*/**            if ( len_got > 0 )**                memmove( buffer, buffer + 9, sizeof(buffer) - 9 );**            else if ( len_got < 0 )                  /\* Move log data to start of buffer*/**            {**                printf("Incomplete record header in %s, record %d (type %d)\n",filename, hdr.log_nmbr,hdr.log_type);**                continue;**            }{*}*            :**            :**            :**            :* {panel}]]> 2162861 2195628 tm_year + 1900)) + tp->tm_yday + 1;            else                itime = (1000 * tp->tm_year) + tp->tm_yday + 1;             dtime = tp->tm_yday + 1.0 + (double)((3600 * tp->tm_hour)                                    + (60 * tp->tm_min) + tp->tm_sec) / 86400.0;             len_got = cc - 9;                        /\* compute amount of log data gotten*/            if ( len_got > 0 )                memmove( buffer, buffer + 9, sizeof(buffer) - 9 );            else if ( len_got < 0 )                  /\* Move log data to start of buffer*/            {                printf("Incomplete record header in %s, record %d (type %d)\n",filename, hdr.log_nmbr,hdr.log_type);                continue;            }            :            :            :            :]]> 2162862 2195629 tm_year + 1900)) + tp->tm_yday + 1;            else                itime = (1000 * tp->tm_year) + tp->tm_yday + 1;             dtime = tp->tm_yday + 1.0 + (double)((3600 * tp->tm_hour)                                    + (60 * tp->tm_min) + tp->tm_sec) / 86400.0;             len_got = cc - 9;                        /\* compute amount of log data gotten*/            if ( len_got > 0 )                memmove( buffer, buffer + 9, sizeof(buffer) - 9 );            else if ( len_got < 0 )                  /\* Move log data to start of buffer*/            {                printf("Incomplete record header in %s, record %d (type %d)\n",filename, hdr.log_nmbr,hdr.log_type);                continue;            }            :            :            :            : {noformat}]]> 2162863 2195630 tm_year + 1900)) + tp->tm_yday + 1;            else                itime = (1000 * tp->tm_year) + tp->tm_yday + 1;             dtime = tp->tm_yday + 1.0 + (double)((3600 * tp->tm_hour)                                    + (60 * tp->tm_min) + tp->tm_sec) / 86400.0;             len_got = cc - 9;                        /\* compute amount of log data gotten*/            if ( len_got > 0 )                memmove( buffer, buffer + 9, sizeof(buffer) - 9 );            else if ( len_got < 0 )                  /\* Move log data to start of buffer*/            {                printf("Incomplete record header in %s, record %d (type %d)\n",filename, hdr.log_nmbr,hdr.log_type);                continue;            }            :            :            :            :\\]]> 2162864 2195623 2162857 2195626 tm_year + 1900)) + tp->tm_yday + 1;            else                itime = (1000 * tp->tm_year) + tp->tm_yday + 1;             dtime = tp->tm_yday + 1.0 + (double)((3600 * tp->tm_hour)                                    + (60 * tp->tm_min) + tp->tm_sec) / 86400.0;             len_got = cc - 9;                        /\* compute amount of log data gotten*/            if ( len_got > 0 )                memmove( buffer, buffer + 9, sizeof(buffer) - 9 );            else if ( len_got < 0 )                  /\* Move log data to start of buffer*/            {                printf("Incomplete record header in %s, record %d (type %d)\n",filename, hdr.log_nmbr,hdr.log_type);                continue;            }            :            :            :            :]]> 2162860 2195619 2162853 2195620 27094) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. Mike McCann (30 October 2007)]]> 2162854 2195621 2162855 2195622 27060) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again - or wait for it to run with the hourly download on tsunami. Mike McCann (30 October 2007)]]> 2162856 9830922 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs to the newly deployed deviceIDs (aka ISI_IDs). Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml wile with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /hosts/tornado_vol0/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed. h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSD ingest will recognize them as new. # +Do not close (set endDates) the existing Test deployment until after the oasisToSSDS has run with the new platform deployment name+ and all new instrument deployments have been created in SSDS_Metadata. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadta, e.g.: {noformat} SELECT id, name, startDate, endDate, ParentID_FK FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') ORDER BY id DESC {noformat} Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file. # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. # When the mooring log message is sent to oasis with the actual time of deployment enter that as the startDate in the database.  All times (DTGs) in the SSDS_Metadata database are GMT. Make sure to set the startDate for the imctd microcat sensors too. This can be done with a query like this where you use the proper dataProduceIDs for the mooring and the imcd: {noformat} SELECT id, name, ParentID_FK, startDate, endDate, nominalDepth FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') AND (ParentID_FK = 31564) OR (ParentID_FK = 31548) ORDER BY id DESC {noformat} # Because of some bug in SSDS ingest the dataContainerTypes of the outputs from the instrument deployments do not get properly assigned the values of 'Stream'.  This needs to be fixed so that the NDBC datatransfers will work. To fix it edit the SSDS_Metadata database starting with a query like this: {noformat} SELECT ssdsdba.DataContainer.* FROM ssdsdba.DataProducer INNER JOIN ssdsdba.DataContainer ON ssdsdba.DataProducer.id = ssdsdba.DataContainer.DataProducerID_FK WHERE (ssdsdba.DataProducer.ParentID_FK = 31548) {noformat} where you use the DataProducerID for the new mooring deployment in the WHERE clause +Change all of the 'File's in the dataContainerType' field to 'Stream's+. Mike McCann (First edit: 30 October 2007, Last updated: 1 May 2009)]]> 9798168 29523973 2 1835089 2016-11-14 10:42:39.540 2016-11-14 10:42:39.540 11829436 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs and the path to the xml files to the newly deployed deviceIDs (aka ISI_IDs) and new xml directory. Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen or JEdit. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml file with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. \[Note: With the M1 - October 2009 deployment we added the individual inductive modem microcats as children of the mooring and also configured a parallel deployment of the TString that produced the same data. With the M2 - April 2010 deployment we configured the individual IM mirocrocats and did not configure a parallel TString deployment. This is a better model for the system and is much easier to configure and produces more easily consumed data by downstream processes such as combineTS.pl.\] With future deployments we will not configure TString. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /mbari/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. \\ For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed (or...as below \-rschramm 4/2010) \\ \\ {noformat} declare @myID as bigint set @myID = 31548 SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE id = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk in ( SELECT id FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL) ORDER BY id {noformat} h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSDS ingest will recognize them as new. # +Do not close (set endDates) the existing Test deployment until after the oasisToSSDS has run with the new platform deployment name+ and all new instrument deployments have been created in SSDS_Metadata. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadata, e.g.: {noformat} SELECT id, name, startDate, endDate, ParentID_FK FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') ORDER BY id DESC {noformat} Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file. # You will also need to edit the DStoNetCDF.pl script to set the new name of the M1 deployment, e.g.: {noformat} # # SSDS mooring name Deployment lookup # my %ssdsMooringDeplNames = ( M0 => 'M0', M1 => 'M1 - October 2009', M1Test => 'Test M1 - October 2009', M2 => 'M2 - April 2009', {noformat} # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. # When the mooring log message is sent to oasis with the actual time of deployment enter that as the startDate in the database.  All times (DTGs) in the SSDS_Metadata database are GMT. Make sure to set the startDate for the imctd microcat sensors too. This can be done with a query like this where you use the proper dataProduceIDs for the mooring and the imcd: {noformat} SELECT id, name, ParentID_FK, startDate, endDate, nominalDepth FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') AND (ParentID_FK = 31564) OR (ParentID_FK = 31548) ORDER BY id DESC {noformat} # Because of some bug in SSDS ingest the dataContainerTypes of the outputs from the instrument deployments do not get properly assigned the values of 'Stream'.  This needs to be fixed so that the NDBC datatransfers will work. To fix it edit the SSDS_Metadata database starting with a query like this: {noformat} SELECT ssdsdba.DataContainer.* FROM ssdsdba.DataProducer INNER JOIN ssdsdba.DataContainer ON ssdsdba.DataProducer.id = ssdsdba.DataContainer.DataProducerID_FK WHERE (ssdsdba.DataProducer.ParentID_FK = 31548) {noformat} where you use the DataProducerID for the new mooring deployment in the WHERE clause. ==> Change all of the 'File's in the dataContainerType' field to 'Stream's. Mike McCann (First edit: 30 October 2007, Last updated: 1 May 2009)]]> 11796677 11829442 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs and the path to the xml files to the newly deployed deviceIDs (aka ISI_IDs) and new xml directory. Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen or JEdit. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml file with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. \[Note: With the M1 - October 2009 deployment we added the individual inductive modem microcats as children of the mooring and also configured a parallel deployment of the TString that produced the same data. With the M2 - April 2010 deployment we configured the individual IM mirocrocats and did not configure a parallel TString deployment. This is a better model for the system and is much easier to configure and produces more easily consumed data by downstream processes such as combineTS.pl.\] With future deployments we will not configure TString. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /mbari/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. \\ For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed (or...as below \-rschramm 4/2010) \\ \\ {noformat} declare @myID as bigint set @myID = 31548 SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE id = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk in ( SELECT id FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL) ORDER BY id {noformat} h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSDS ingest will recognize them as new. # +Do not close (set endDates) the existing Test deployment until after the oasisToSSDS has run with the new platform deployment name+ and all new instrument deployments have been created in SSDS_Metadata. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadata, e.g.: {noformat} SELECT id, name, startDate, endDate, ParentID_FK FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') ORDER BY id DESC {noformat} Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file. # You will also need to edit the DStoNetCDF.pl script to set the new name of the M1 deployment, e.g.: {noformat} # # SSDS mooring name Deployment lookup # my %ssdsMooringDeplNames = ( M0 => 'M0', M1 => 'M1 - October 2009', M1Test => 'Test M1 - October 2009', M2 => 'M2 - April 2009', {noformat} # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. # When the mooring log message is sent to oasis with the actual time of deployment enter that as the startDate in the database.  All times (DTGs) in the SSDS_Metadata database are GMT. Make sure to set the startDate for the imctd microcat sensors too. This can be done with a query like this where you use the proper dataProduceIDs for the mooring and the imcd: {noformat} SELECT id, name, ParentID_FK, startDate, endDate, nominalDepth FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') AND (ParentID_FK = 31564) OR (ParentID_FK = 31548) ORDER BY id DESC {noformat} # Because of some bug in SSDS ingest the dataContainerTypes of the outputs from the instrument deployments do not get properly assigned the values of 'Stream'.  This needs to be fixed so that the NDBC datatransfers will work. To fix it edit the SSDS_Metadata database starting with a query like this: {noformat} SELECT ssdsdba.DataContainer.* FROM ssdsdba.DataProducer INNER JOIN ssdsdba.DataContainer ON ssdsdba.DataProducer.id = ssdsdba.DataContainer.DataProducerID_FK WHERE (ssdsdba.DataProducer.ParentID_FK = 31548) {noformat} where you use the DataProducerID for the new mooring deployment in the WHERE clause. ==> Change all of the 'File's in the dataContainerType' field to 'Stream's. h2. C. Procedures to be done after the mooring turn h4. Set up download info deployment and reporting Though not a real instrument, we configure a virtual 'dlinfo' instrument for the download scripts to attach download statistics data. We re-use the same device IDs for the M1 and M2 moorings, so it's best to configure this after the new mooring is out and all of those deployments have been closed. As the data are delivered "out of band" from the typical OASIS instruments we need to create a deployment for dlinfo instrument by hand. The most direct way is to use the createDuplicateDeepDeployment service call. For example, to duplicate the 2009 M2 dlinfo deployment for the 2010 M2 deployment this call was executed: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer|id=34496&p2Type=Date&p2Value=2010-04-03T22:00:00Z&p3Type=boolean&p3Value=false&p4Type=Date&p4Value=2010-04-03T23:00:00Z&p5Type=String&p5Value=getM2-download&p6Type=String&p6Value=&delimiter=|. Of course you will need to adjust the DataProducer ID Date values for the new deployment you are creating. Here is the Key to the parameters: Mike McCann (First edit: 30 October 2007, Last updated: 1 May 2009)]]> 11796683 18645514 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs and the path to the xml files to the newly deployed deviceIDs (aka ISI_IDs) and new xml directory. Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen or JEdit. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml file with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. \[Note: With the M1 - October 2009 deployment we added the individual inductive modem microcats as children of the mooring and also configured a parallel deployment of the TString that produced the same data. With the M2 - April 2010 deployment we configured the individual IM mirocrocats and did not configure a parallel TString deployment. This is a better model for the system and is much easier to configure and produces more easily consumed data by downstream processes such as combineTS.pl.\] With future deployments we will not configure TString. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|SSDS:Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[SSDS:main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /mbari/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. \\ For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed (or...as below \-rschramm 4/2010) \\ \\ {noformat} declare @myID as bigint set @myID = 31548 SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE id = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk in ( SELECT id FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL) ORDER BY id {noformat} h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSDS ingest will recognize them as new. # +Do not close (set endDates) the existing Test deployment until after the oasisToSSDS has run with the new platform deployment name+ and all new instrument deployments have been created in SSDS_Metadata. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadata, e.g.: {noformat} SELECT id, name, startDate, endDate, ParentID_FK FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') ORDER BY id DESC {noformat} Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file. # You will also need to edit the DStoNetCDF.pl script to set the new name of the M1 deployment, e.g.: {noformat} # # SSDS mooring name Deployment lookup # my %ssdsMooringDeplNames = ( M0 => 'M0', M1 => 'M1 - October 2009', M1Test => 'Test M1 - October 2009', M2 => 'M2 - April 2009', {noformat} # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. # When the mooring log message is sent to oasis with the actual time of deployment enter that as the startDate in the database.  All times (DTGs) in the SSDS_Metadata database are GMT. Make sure to set the startDate for the imctd microcat sensors too. This can be done with a query like this where you use the proper dataProduceIDs for the mooring and the imcd: {noformat} SELECT id, name, ParentID_FK, startDate, endDate, nominalDepth FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') AND (ParentID_FK = 31564) OR (ParentID_FK = 31548) ORDER BY id DESC {noformat} # Because of some bug in SSDS ingest the dataContainerTypes of the outputs from the instrument deployments do not get properly assigned the values of 'Stream'.  This needs to be fixed so that the NDBC datatransfers will work. To fix it edit the SSDS_Metadata database starting with a query like this: {noformat} SELECT ssdsdba.DataContainer.* FROM ssdsdba.DataProducer INNER JOIN ssdsdba.DataContainer ON ssdsdba.DataProducer.id = ssdsdba.DataContainer.DataProducerID_FK WHERE (ssdsdba.DataProducer.ParentID_FK = 31548) {noformat} where you use the DataProducerID for the new mooring deployment in the WHERE clause. ==> Change all of the 'File's in the dataContainerType' field to 'Stream's. h2. C. Procedures to be done after the mooring turn h4. Set up download info deployment and reporting Though not a real instrument, we configure a virtual 'dlinfo' instrument for the download scripts to attach download statistics data. We re-use the same device IDs for the M1 and M2 moorings, so it's best to configure this after the new mooring is out and all of those deployments have been closed. As the data are delivered "out of band" from the typical OASIS instruments we need to create a deployment for dlinfo instrument by hand. # The most direct way is to use the createDuplicateDeepDeployment service call. For example, to duplicate the 2009 M2 dlinfo deployment for the 2010 M2 deployment this call was executed: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer\|id=*34496*&p2Type=Date&p2Value=*2010-04-03T22:00:00Z*&p3Type=boolean&p3Value=false&p4Type=Date&p4Value=*2010-04-03T23:00:00Z*&p5Type=String&p5Value=getM2-download&p6Type=String&p6Value=&delimiter=\|. Of course you will need to adjust the DataProducer ID Date values for the new deployment you are creating. Here is the Key to the parameters: {noformat} Key: ---- http://localhost:8080/servlet/MetadataAccessServlet ?responseType=text &delimiter=| &objectToInvokeOn=DataProducerAccess &method=createDuplicateDeepDeployment &p1Type=DataProducer &p1Value=DataProducer|id=XXXX (XXXX is the ID of the deployment to copy) &p2Type=Date &p2Value=XXXXXX (XXXXXX is the start date of the new copy in XML format YYYY-MM-DDTHH:MM:SSZ) &p3Type=boolean &p3Value=(true|false) (this is to indicate if you want the original deployment to be closed) &p4Type=Date &p4Value=XXXXXX (XXXXXX is the end date for the original deployment (if p3Value is true)) &p5Type=String &p5Value=XXXXXX (XXXXXX is the DataProducer name of the new DataProducer) &p6Type=String &p6Value=XXXXXX (XXXXXX is the base URL to use for the new DataContainers that will be created). &delimiter=| {noformat}\\ Executing the createDuplicateDeepDeployment service call will return an ID for the new deployment. # Edit the new deployment record in the DataProducer table to adjust it's parentID_FK to be for the new mooring. While there edit times and name as appropriate. # Edit the getM? script on tsunami to use the proper device and parent ID (this will be the device ID of the torroid of the mooring deployment) and make sure that the '/oasis/bin/ssdsSubmit.pl $deviceId $parentId "$starttime_es,$endtime_es,$filesize,$rtnsts"' line in the script is configured to run. Here's another example for the October 2010 M1 turn: {noformat} http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&\ p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer|id=35591&p2Type=Date&p2Value=2010-10-27T21:00:00Z&p3Type=boolean&p3Value=false&p4Type=Date&\ p4Value=2010-q0-27T20:30:00Z&p5Type=String&p5Value=getM1-download&p6Type=String&p6Value=&delimiter=| (This returned a new DataProducerID of 41556. The ParentID_FK for this record in the DataProducer table was then changed from 35388 to 41197 and the endDate cleared to null. Also, with the change to alternating deviceIDs implemented in 2010 the deviceID_FK was changed from 1698 to 1768 for this M1 deployment. {noformat} N.B. A rotating scheme for the virtual device IDs was implemented in 2010. This permits the steps described above to be executed prior to the mooring turn during the extensive dock-side test period. h4. Enable processing for other "virtual" devices # Turn on ClockSync processing. We also use a virtual device ID for these data. Simply uncomment the line for it in the ssds.cfg file. # Turn on ISUS processing. Simply uncomment the line for it in the ssds.cfg file. (Device ID re-used from previous deployment.) h4. Make all of the child instrument deployment start times the same as the mooring start time When SSDS receives a packet from an instrument that is not currently deployed it will create a Deployment record (in the DataProducer table) with a startDate that is set to the time of the first record received. As the mooring starts up all of the instrument deployments will have different start times based on when they each first saw data. This can present problems for the data processing that follows, especially for the jobs that aggregate the microcat data into a single ZT file that is used to produce the contour temperature and salinity wind stick plots. To prevent these problems it's best to edit the startDates of the child instrument deployments so that they are all the same. This is currently most easily done through Enterprise Manager with a query like below (this is for the 201010 M1 deployment) to get all the child instrument deployments: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 41197) ORDER BY startDate {noformat} Then copy and paste the datetime string from one cell to the next. The times are GMT. h4. Cycle links to previous deployment # Edit previous.html file in /mbari/ssdsdata/deployments to add a line for the new deployment and add the end date and archive url for the just closed deployment, e.g.: {noformat}
  • 200904 2009-04-29 to 2010-04-03
  • 201004 2010-04-03 to present
    • {noformat}\\ # Add lines to DEPLOYMENTS file in dev/DPforSSDS/cimt/ for the just closed deployment, e.g.: {noformat} DStoNetCDF.pl -mooring M2 -deployment "M2 - April 2009" -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org -outputDir \ /mbari/ssdsdata/deployments -procClosed -verbose combineM.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments combineTS.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments combineAll.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments and /bin/cp /mbari/ssdsdata/deployments/m2/200904/OS_*.nc /mbari/FTP/pub/OceanSITES {noformat}\\ # Execute these lines to generate "closed deployment" data products and web pages and for submission to the OceanSITES GDAC # Confirm that the processing executed properly. Sometimes mangled timestamp data weasels its way into the data stream causing instrument netcdf files to be named with bogus start dates. (This can also happen during a deployment and is one of the maintenance tasks one should follow to keep the data flowing to where it needs to go.) The combine__.pl scripts will then create the OceanSITES formatted files encompassing the dates of all the instrument netcdf files. Incorrect instrument file names will create incorrect OS_MBARI* file names causing CenCOOS and NDBC/Ifremer to complain about the correctly named file not being updated with new data. The fix involves a purging of the bad files, double checking the metadata in SSDS and reprocessing, with perhaps additional checks in the instrument processing perl code to skip over bad records. This is best done at the Unix command line by cd'ing to the deployment directory, e.g. {noformat} cd /mbari/deployments/m1/201010 ls -lrt {noformat} and removing files that have bogus dates that do not represent the deployment start date. You should also remove all the files directories and files in the gifs/ subdirectory. These will all get recreated when you run DStoNetCDF.pl and the combine__.pl scripts. With luck simply removing the bad files and rerunning the processing scripts will fix things. If not, identify where the bad dates are coming from and fix as appropriate. Mike McCann (First edit: 30 October 2007, Last updated: 1 May 2009)]]>     18580006     6226304 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. # Edit the mooring .cfg file and changed the instrument deviceIDs to the newly deployed deviceIDs (aka ISI_IDs). Refer to email from OSG for the correct numbers, e.g.: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files. Remove any time or location information and check back into CVS. Copy the files into the _yyyy_/xml subdirectory. The one exception is the platform deployment XML file; its Deployment element should contain name and startDate attributes so that the parent deployment may be found in the Explorer application, e.g.: {noformat} {noformat} # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml wile with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. h2. B. Procedure for doing the actual production mooring turn So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. The steps to do this are outlined below. \[As of November 2007, there are several issues with making this a fool-proof set of instructions; therefore, for now they will serve as documentation for tasks  in the 2008 SSDS Hardening project.\] \\ # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id > 27060) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the instrument swap procedure and doing a deep delete on the wrong instrument deployment. If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: <251 elvis.shore.mbari.org /u/ssdsadmin> dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02 \- or wait for it to run with the hourly download on tsunami. #   #   #   Mike McCann (30 October 2007)]]> 6259086 6226300 ) on the wrong instrument deployment. # Test by ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. h2. B. Procedure for doing the actual production mooring turn So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. The steps to do this are outlined below. \[As of November 2007, there are several issues with making this a fool-proof set of instructions; therefore, for now they will serve as documentation for tasks  in the 2008 SSDS Hardening project.\] \\ # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id > 27060) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the instrument swap procedure and doing a deep delete on the wrong instrument deployment. If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: <251 elvis.shore.mbari.org /u/ssdsadmin> dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02 \- or wait for it to run with the hourly download on tsunami. #   #   #   Mike McCann (30 October 2007)]]> 6259082 6226302 .) The new directory will have subdirectories named data, cfg, and xml. # Edit the mooring .cfg file and changed the instrument deviceIDs to the newly deployed deviceIDs (aka ISI_IDs). Refer to email from OSG for the correct numbers, e.g.: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files. Remove any time or location information and check back into CVS. Copy the files into the _yyyy_/xml subdirectory. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml wile with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. h2. B. Procedure for doing the actual production mooring turn So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. The steps to do this are outlined below. \[As of November 2007, there are several issues with making this a fool-proof set of instructions; therefore, for now they will serve as documentation for tasks  in the 2008 SSDS Hardening project.\] \\ # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id > 27060) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the instrument swap procedure and doing a deep delete on the wrong instrument deployment. If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: <251 elvis.shore.mbari.org /u/ssdsadmin> dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02 \- or wait for it to run with the hourly download on tsunami. #   #   #   Mike McCann (30 October 2007)]]> 6259084 2195465 2162698 10389056 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs to the newly deployed deviceIDs (aka ISI_IDs). Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml wile with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /hosts/tornado_vol0/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed. h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSD ingest will recognize them as new. # +Do not close (set endDates) the existing Test deployment until after the oasisToSSDS has run with the new platform deployment name+ and all new instrument deployments have been created in SSDS_Metadata. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadta, e.g.: {noformat} SELECT id, name, startDate, endDate, ParentID_FK FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') ORDER BY id DESC {noformat} Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file. # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. # When the mooring log message is sent to oasis with the actual time of deployment enter that as the startDate in the database.  All times (DTGs) in the SSDS_Metadata database are GMT. Make sure to set the startDate for the imctd microcat sensors too. This can be done with a query like this where you use the proper dataProduceIDs for the mooring and the imcd: {noformat} SELECT id, name, ParentID_FK, startDate, endDate, nominalDepth FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') AND (ParentID_FK = 31564) OR (ParentID_FK = 31548) ORDER BY id DESC {noformat} # Because of some bug in SSDS ingest the dataContainerTypes of the outputs from the instrument deployments do not get properly assigned the values of 'Stream'.  This needs to be fixed so that the NDBC datatransfers will work. To fix it edit the SSDS_Metadata database starting with a query like this: {noformat} SELECT ssdsdba.DataContainer.* FROM ssdsdba.DataProducer INNER JOIN ssdsdba.DataContainer ON ssdsdba.DataProducer.id = ssdsdba.DataContainer.DataProducerID_FK WHERE (ssdsdba.DataProducer.ParentID_FK = 31548) {noformat} where you use the DataProducerID for the new mooring deployment in the WHERE clause. ==> Change all of the 'File's in the dataContainerType' field to 'Stream's. Mike McCann (First edit: 30 October 2007, Last updated: 1 May 2009)]]> 10356341 10944554 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs to the newly deployed deviceIDs (aka ISI_IDs). Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml wile with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /mbari/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed. h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSDS ingest will recognize them as new. # +Do not close (set endDates) the existing Test deployment until after the oasisToSSDS has run with the new platform deployment name+ and all new instrument deployments have been created in SSDS_Metadata. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadata, e.g.: {noformat} SELECT id, name, startDate, endDate, ParentID_FK FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') ORDER BY id DESC {noformat} Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file. # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. # When the mooring log message is sent to oasis with the actual time of deployment enter that as the startDate in the database.  All times (DTGs) in the SSDS_Metadata database are GMT. Make sure to set the startDate for the imctd microcat sensors too. This can be done with a query like this where you use the proper dataProduceIDs for the mooring and the imcd: {noformat} SELECT id, name, ParentID_FK, startDate, endDate, nominalDepth FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') AND (ParentID_FK = 31564) OR (ParentID_FK = 31548) ORDER BY id DESC {noformat} # Because of some bug in SSDS ingest the dataContainerTypes of the outputs from the instrument deployments do not get properly assigned the values of 'Stream'.  This needs to be fixed so that the NDBC datatransfers will work. To fix it edit the SSDS_Metadata database starting with a query like this: {noformat} SELECT ssdsdba.DataContainer.* FROM ssdsdba.DataProducer INNER JOIN ssdsdba.DataContainer ON ssdsdba.DataProducer.id = ssdsdba.DataContainer.DataProducerID_FK WHERE (ssdsdba.DataProducer.ParentID_FK = 31548) {noformat} where you use the DataProducerID for the new mooring deployment in the WHERE clause. ==> Change all of the 'File's in the dataContainerType' field to 'Stream's. Mike McCann (First edit: 30 October 2007, Last updated: 1 May 2009)]]> 10911787 10944551 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs to the newly deployed deviceIDs (aka ISI_IDs). Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml wile with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /mbari/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed. h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSD ingest will recognize them as new. # +Do not close (set endDates) the existing Test deployment until after the oasisToSSDS has run with the new platform deployment name+ and all new instrument deployments have been created in SSDS_Metadata. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadta, e.g.: {noformat} SELECT id, name, startDate, endDate, ParentID_FK FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') ORDER BY id DESC {noformat} Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file. # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. # When the mooring log message is sent to oasis with the actual time of deployment enter that as the startDate in the database.  All times (DTGs) in the SSDS_Metadata database are GMT. Make sure to set the startDate for the imctd microcat sensors too. This can be done with a query like this where you use the proper dataProduceIDs for the mooring and the imcd: {noformat} SELECT id, name, ParentID_FK, startDate, endDate, nominalDepth FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') AND (ParentID_FK = 31564) OR (ParentID_FK = 31548) ORDER BY id DESC {noformat} # Because of some bug in SSDS ingest the dataContainerTypes of the outputs from the instrument deployments do not get properly assigned the values of 'Stream'.  This needs to be fixed so that the NDBC datatransfers will work. To fix it edit the SSDS_Metadata database starting with a query like this: {noformat} SELECT ssdsdba.DataContainer.* FROM ssdsdba.DataProducer INNER JOIN ssdsdba.DataContainer ON ssdsdba.DataProducer.id = ssdsdba.DataContainer.DataProducerID_FK WHERE (ssdsdba.DataProducer.ParentID_FK = 31548) {noformat} where you use the DataProducerID for the new mooring deployment in the WHERE clause. ==> Change all of the 'File's in the dataContainerType' field to 'Stream's. Mike McCann (First edit: 30 October 2007, Last updated: 1 May 2009)]]> 10911784 10944552 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs to the newly deployed deviceIDs (aka ISI_IDs). Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml wile with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /mbari/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed. h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSD ingest will recognize them as new. # +Do not close (set endDates) the existing Test deployment until after the oasisToSSDS has run with the new platform deployment name+ and all new instrument deployments have been created in SSDS_Metadata. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadata, e.g.: {noformat} SELECT id, name, startDate, endDate, ParentID_FK FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') ORDER BY id DESC {noformat} Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file. # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. # When the mooring log message is sent to oasis with the actual time of deployment enter that as the startDate in the database.  All times (DTGs) in the SSDS_Metadata database are GMT. Make sure to set the startDate for the imctd microcat sensors too. This can be done with a query like this where you use the proper dataProduceIDs for the mooring and the imcd: {noformat} SELECT id, name, ParentID_FK, startDate, endDate, nominalDepth FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') AND (ParentID_FK = 31564) OR (ParentID_FK = 31548) ORDER BY id DESC {noformat} # Because of some bug in SSDS ingest the dataContainerTypes of the outputs from the instrument deployments do not get properly assigned the values of 'Stream'.  This needs to be fixed so that the NDBC datatransfers will work. To fix it edit the SSDS_Metadata database starting with a query like this: {noformat} SELECT ssdsdba.DataContainer.* FROM ssdsdba.DataProducer INNER JOIN ssdsdba.DataContainer ON ssdsdba.DataProducer.id = ssdsdba.DataContainer.DataProducerID_FK WHERE (ssdsdba.DataProducer.ParentID_FK = 31548) {noformat} where you use the DataProducerID for the new mooring deployment in the WHERE clause. ==> Change all of the 'File's in the dataContainerType' field to 'Stream's. Mike McCann (First edit: 30 October 2007, Last updated: 1 May 2009)]]> 10911785 33161222 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs and the path to the xml files to the newly deployed deviceIDs (aka ISI_IDs) and new xml directory. Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen or JEdit. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml file with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. \[Note: With the M1 - October 2009 deployment we added the individual inductive modem microcats as children of the mooring and also configured a parallel deployment of the TString that produced the same data. With the M2 - April 2010 deployment we configured the individual IM mirocrocats and did not configure a parallel TString deployment. This is a better model for the system and is much easier to configure and produces more easily consumed data by downstream processes such as combineTS.pl.\] With future deployments we will not configure TString. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|SSDS:Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[SSDS:main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /mbari/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. \\ For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed (or...as below \-rschramm 4/2010) \\ \\ {noformat} declare @myID as bigint set @myID = 31548 SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE id = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk in ( SELECT id FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL) ORDER BY id {noformat} h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSDS ingest will recognize them as new. # +Do not close (set endDates) the existing Test deployment until after the oasisToSSDS has run with the new platform deployment name+ and all new instrument deployments have been created in SSDS_Metadata. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadata, e.g.: {noformat} SELECT id, name, startDate, endDate, ParentID_FK FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') ORDER BY id DESC {noformat} Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file. # You will also need to edit the DStoNetCDF.pl script to set the new name of the M1 deployment, e.g.: {noformat} # # SSDS mooring name Deployment lookup # my %ssdsMooringDeplNames = ( M0 => 'M0', M1 => 'M1 - October 2009', M1Test => 'Test M1 - October 2009', M2 => 'M2 - April 2009', {noformat} # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. # When the mooring log message is sent to oasis with the actual time of deployment enter that as the startDate in the database.  All times (DTGs) in the SSDS_Metadata database are GMT. Make sure to set the startDate for the imctd microcat sensors too. This can be done with a query like this where you use the proper dataProduceIDs for the mooring and the imcd: {noformat} SELECT id, name, ParentID_FK, startDate, endDate, nominalDepth FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') AND (ParentID_FK = 31564) OR (ParentID_FK = 31548) ORDER BY id DESC {noformat} # Because of some bug in SSDS ingest the dataContainerTypes of the outputs from the instrument deployments do not get properly assigned the values of 'Stream'.  This needs to be fixed so that the NDBC datatransfers will work. To fix it edit the SSDS_Metadata database starting with a query like this: {noformat} SELECT ssdsdba.DataContainer.* FROM ssdsdba.DataProducer INNER JOIN ssdsdba.DataContainer ON ssdsdba.DataProducer.id = ssdsdba.DataContainer.DataProducerID_FK WHERE (ssdsdba.DataProducer.ParentID_FK = 31548) {noformat} where you use the DataProducerID for the new mooring deployment in the WHERE clause. ==> Change all of the 'File's in the dataContainerType' field to 'Stream's. h2. C. Procedures to be done after the mooring turn h4. Set up download info deployment and reporting Though not a real instrument, we configure a virtual 'dlinfo' instrument for the download scripts to attach download statistics data. We re-use the same device IDs for the M1 and M2 moorings, so it's best to configure this after the new mooring is out and all of those deployments have been closed. As the data are delivered "out of band" from the typical OASIS instruments we need to create a deployment for dlinfo instrument by hand. # The most direct way is to use the createDuplicateDeepDeployment service call. For example, to duplicate the 2009 M2 dlinfo deployment for the 2010 M2 deployment this call was executed: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer\|id=*34496*&p2Type=Date&p2Value=*2010-04-03T22:00:00Z*&p3Type=boolean&p3Value=false&p4Type=Date&p4Value=*2010-04-03T23:00:00Z*&p5Type=String&p5Value=getM2-download&p6Type=String&p6Value=&delimiter=\|. Of course you will need to adjust the DataProducer ID Date values for the new deployment you are creating. Here is the Key to the parameters: {noformat} Key: ---- http://localhost:8080/servlet/MetadataAccessServlet ?responseType=text &delimiter=| &objectToInvokeOn=DataProducerAccess &method=createDuplicateDeepDeployment &p1Type=DataProducer &p1Value=DataProducer|id=XXXX (XXXX is the ID of the deployment to copy) &p2Type=Date &p2Value=XXXXXX (XXXXXX is the start date of the new copy in XML format YYYY-MM-DDTHH:MM:SSZ) &p3Type=boolean &p3Value=(true|false) (this is to indicate if you want the original deployment to be closed) &p4Type=Date &p4Value=XXXXXX (XXXXXX is the end date for the original deployment (if p3Value is true)) &p5Type=String &p5Value=XXXXXX (XXXXXX is the DataProducer name of the new DataProducer) &p6Type=String &p6Value=XXXXXX (XXXXXX is the base URL to use for the new DataContainers that will be created). &delimiter=| {noformat}\\ Executing the createDuplicateDeepDeployment service call will return an ID for the new deployment. # Edit the new deployment record in the DataProducer table to adjust it's parentID_FK to be for the new mooring. While there edit times and name as appropriate. # Edit the getM? script on tsunami to use the proper device and parent ID (this will be the device ID of the torroid of the mooring deployment) and make sure that the '/oasis/bin/ssdsSubmit.pl $deviceId $parentId "$starttime_es,$endtime_es,$filesize,$rtnsts"' line in the script is configured to run. Here's another example for the October 2010 M1 turn: {noformat} http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&\ p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer|id=35591&p2Type=Date&p2Value=2010-10-27T21:00:00Z&p3Type=boolean&p3Value=false&p4Type=Date&\ p4Value=2010-q0-27T20:30:00Z&p5Type=String&p5Value=getM1-download&p6Type=String&p6Value=&delimiter=| (This returned a new DataProducerID of 41556. The ParentID_FK for this record in the DataProducer table was then changed from 35388 to 41197 and the endDate cleared to null. Also, with the change to alternating deviceIDs implemented in 2010 the deviceID_FK was changed from 1698 to 1768 for this M1 deployment. {noformat} N.B. A rotating scheme for the virtual device IDs was implemented in 2010. This permits the steps described above to be executed prior to the mooring turn during the extensive dock-side test period. h4. Enable processing for other "virtual" devices # Turn on ClockSync processing. We also use a virtual device ID for these data. Simply uncomment the line for it in the ssds.cfg file. # Turn on ISUS processing. Simply uncomment the line for it in the ssds.cfg file. (Device ID re-used from previous deployment.) h4. Make all of the child instrument deployment start times the same as the mooring start time When SSDS receives a packet from an instrument that is not currently deployed it will create a Deployment record (in the DataProducer table) with a startDate that is set to the time of the first record received. As the mooring starts up all of the instrument deployments will have different start times based on when they each first saw data. This can present problems for the data processing that follows, especially for the jobs that aggregate the microcat data into a single ZT file that is used to produce the contour temperature and salinity wind stick plots. To prevent these problems it's best to edit the startDates of the child instrument deployments so that they are all the same. This is currently most easily done through Enterprise Manager with a query like below (this is for the 201010 M1 deployment) to get all the child instrument deployments: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 41197) ORDER BY startDate {noformat} Then copy and paste the datetime string from one cell to the next. The times are GMT. h4. Cycle links to previous deployment # Edit previous.html file in /mbari/ssdsdata/deployments to add a line for the new deployment and add the end date and archive url for the just closed deployment, e.g.: {noformat}
  • 200904 2009-04-29 to 2010-04-03
  • 201004 2010-04-03 to present
    • {noformat}\\ # Add lines to DEPLOYMENTS file in dev/DPforSSDS/cimt/ for the just closed deployment, e.g.: {noformat} DStoNetCDF.pl -mooring M2 -deployment "M2 - April 2009" -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org -outputDir \ /mbari/ssdsdata/deployments -procClosed -verbose combineM.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments combineTS.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments combineAll.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments and /bin/cp /mbari/ssdsdata/deployments/m2/200904/OS_*.nc /mbari/FTP/pub/OceanSITES {noformat}\\ # Execute these lines to generate "closed deployment" data products and web pages and for submission to the OceanSITES GDAC # Confirm that the processing executed properly. Sometimes mangled timestamp data weasels its way into the data stream causing instrument netcdf files to be named with bogus start dates. (This can also happen during a deployment and is one of the maintenance tasks one should follow to keep the data flowing to where it needs to go.) The combine__.pl scripts will then create the OceanSITES formatted files encompassing the dates of all the instrument netcdf files. Incorrect instrument file names will create incorrect OS_MBARI* file names causing CenCOOS and NDBC/Ifremer to complain about the correctly named file not being updated with new data. The fix involves a purging of the bad files, double checking the metadata in SSDS and reprocessing, with perhaps additional checks in the instrument processing perl code to skip over bad records. This is best done at the Unix command line by cd'ing to the deployment directory, e.g. {noformat} cd /mbari/deployments/m1/201010 ls -lrt {noformat} and removing files that have bogus dates that do not represent the deployment start date. You should also remove all the directories and files in the gifs/ subdirectory. These will all get recreated when you run DStoNetCDF.pl and the combine__.pl scripts. With luck, simply removing the bad files and rerunning the processing scripts will fix things. If not, identify where the bad dates are coming from and fix as appropriate. Mike McCann (First edit: 30 October 2007, Last updated: 1 March 2012)]]>     33095686     30867463 6422536 1835089 2019-01-07 16:00:43.370 2019-01-07 16:00:43.370 11829444 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs and the path to the xml files to the newly deployed deviceIDs (aka ISI_IDs) and new xml directory. Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen or JEdit. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml file with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. \[Note: With the M1 - October 2009 deployment we added the individual inductive modem microcats as children of the mooring and also configured a parallel deployment of the TString that produced the same data. With the M2 - April 2010 deployment we configured the individual IM mirocrocats and did not configure a parallel TString deployment. This is a better model for the system and is much easier to configure and produces more easily consumed data by downstream processes such as combineTS.pl.\] With future deployments we will not configure TString. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /mbari/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. \\ For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed (or...as below \-rschramm 4/2010) \\ \\ {noformat} declare @myID as bigint set @myID = 31548 SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE id = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk in ( SELECT id FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL) ORDER BY id {noformat} h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSDS ingest will recognize them as new. # +Do not close (set endDates) the existing Test deployment until after the oasisToSSDS has run with the new platform deployment name+ and all new instrument deployments have been created in SSDS_Metadata. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadata, e.g.: {noformat} SELECT id, name, startDate, endDate, ParentID_FK FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') ORDER BY id DESC {noformat} Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file. # You will also need to edit the DStoNetCDF.pl script to set the new name of the M1 deployment, e.g.: {noformat} # # SSDS mooring name Deployment lookup # my %ssdsMooringDeplNames = ( M0 => 'M0', M1 => 'M1 - October 2009', M1Test => 'Test M1 - October 2009', M2 => 'M2 - April 2009', {noformat} # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. # When the mooring log message is sent to oasis with the actual time of deployment enter that as the startDate in the database.  All times (DTGs) in the SSDS_Metadata database are GMT. Make sure to set the startDate for the imctd microcat sensors too. This can be done with a query like this where you use the proper dataProduceIDs for the mooring and the imcd: {noformat} SELECT id, name, ParentID_FK, startDate, endDate, nominalDepth FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') AND (ParentID_FK = 31564) OR (ParentID_FK = 31548) ORDER BY id DESC {noformat} # Because of some bug in SSDS ingest the dataContainerTypes of the outputs from the instrument deployments do not get properly assigned the values of 'Stream'.  This needs to be fixed so that the NDBC datatransfers will work. To fix it edit the SSDS_Metadata database starting with a query like this: {noformat} SELECT ssdsdba.DataContainer.* FROM ssdsdba.DataProducer INNER JOIN ssdsdba.DataContainer ON ssdsdba.DataProducer.id = ssdsdba.DataContainer.DataProducerID_FK WHERE (ssdsdba.DataProducer.ParentID_FK = 31548) {noformat} where you use the DataProducerID for the new mooring deployment in the WHERE clause. ==> Change all of the 'File's in the dataContainerType' field to 'Stream's. h2. C. Procedures to be done after the mooring turn h4. Set up download info deployment and reporting Though not a real instrument, we configure a virtual 'dlinfo' instrument for the download scripts to attach download statistics data. We re-use the same device IDs for the M1 and M2 moorings, so it's best to configure this after the new mooring is out and all of those deployments have been closed. As the data are delivered "out of band" from the typical OASIS instruments we need to create a deployment for dlinfo instrument by hand. # The most direct way is to use the createDuplicateDeepDeployment service call. For example, to duplicate the 2009 M2 dlinfo deployment for the 2010 M2 deployment this call was executed: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer\|id=34496&p2Type=Date&p2Value=2010-04-03T22:00:00Z&p3Type=boolean&p3Value=false&p4Type=Date&p4Value=2010-04-03T23:00:00Z&p5Type=String&p5Value=getM2-download&p6Type=String&p6Value=&delimiter=\|. Of course you will need to adjust the DataProducer ID Date values for the new deployment you are creating. Here is the Key to the parameters: {noformat} Key: ---- http://localhost:8080/servlet/MetadataAccessServlet ?responseType=text &delimiter=| &objectToInvokeOn=DataProducerAccess &method=createDuplicateDeepDeployment &p1Type=DataProducer &p1Value=DataProducer|id=XXXX (XXXX is the ID of the deployment to copy) &p2Type=Date &p2Value=XXXXXX (XXXXXX is the start date of the new copy in XML format YYYY-MM-DDTHH:MM:SSZ) &p3Type=boolean &p3Value=(true|false) (this is to indicate if you want the original deployment to be closed) &p4Type=Date &p4Value=XXXXXX (XXXXXX is the end date for the original deployment (if p3Value is true)) &p5Type=String &p5Value=XXXXXX (XXXXXX is the DataProducer name of the new DataProducer) &p6Type=String &p6Value=XXXXXX (XXXXXX is the base URL to use for the new DataContainers that will be created). &delimiter=| {noformat}\\ Executing the createDuplicateDeepDeployment service call will return an ID for the new deployment. # Edit the new deployment record in the DataProducer table to adjust it's parentID_FK to be for the new mooring. While there adit times and name as appropriate. # Edit the getM? script on tsunami to use the proper device and parent ID (this will be the device ID of the torroid of the mooring deployment) and make sure that the '/oasis/bin/ssdsSubmit.pl $deviceId $parentId "$starttime_es,$endtime_es,$filesize,$rtnsts"' line in the script is configured to run. h4. Turn on ISUS processing. We use a virtual ISUS device ID Mike McCann (First edit: 30 October 2007, Last updated: 1 May 2009)]]> 11796685 11829446 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs and the path to the xml files to the newly deployed deviceIDs (aka ISI_IDs) and new xml directory. Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen or JEdit. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml file with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. \[Note: With the M1 - October 2009 deployment we added the individual inductive modem microcats as children of the mooring and also configured a parallel deployment of the TString that produced the same data. With the M2 - April 2010 deployment we configured the individual IM mirocrocats and did not configure a parallel TString deployment. This is a better model for the system and is much easier to configure and produces more easily consumed data by downstream processes such as combineTS.pl.\] With future deployments we will not configure TString. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /mbari/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. \\ For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed (or...as below \-rschramm 4/2010) \\ \\ {noformat} declare @myID as bigint set @myID = 31548 SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE id = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk in ( SELECT id FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL) ORDER BY id {noformat} h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSDS ingest will recognize them as new. # +Do not close (set endDates) the existing Test deployment until after the oasisToSSDS has run with the new platform deployment name+ and all new instrument deployments have been created in SSDS_Metadata. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadata, e.g.: {noformat} SELECT id, name, startDate, endDate, ParentID_FK FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') ORDER BY id DESC {noformat} Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file. # You will also need to edit the DStoNetCDF.pl script to set the new name of the M1 deployment, e.g.: {noformat} # # SSDS mooring name Deployment lookup # my %ssdsMooringDeplNames = ( M0 => 'M0', M1 => 'M1 - October 2009', M1Test => 'Test M1 - October 2009', M2 => 'M2 - April 2009', {noformat} # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. # When the mooring log message is sent to oasis with the actual time of deployment enter that as the startDate in the database.  All times (DTGs) in the SSDS_Metadata database are GMT. Make sure to set the startDate for the imctd microcat sensors too. This can be done with a query like this where you use the proper dataProduceIDs for the mooring and the imcd: {noformat} SELECT id, name, ParentID_FK, startDate, endDate, nominalDepth FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') AND (ParentID_FK = 31564) OR (ParentID_FK = 31548) ORDER BY id DESC {noformat} # Because of some bug in SSDS ingest the dataContainerTypes of the outputs from the instrument deployments do not get properly assigned the values of 'Stream'.  This needs to be fixed so that the NDBC datatransfers will work. To fix it edit the SSDS_Metadata database starting with a query like this: {noformat} SELECT ssdsdba.DataContainer.* FROM ssdsdba.DataProducer INNER JOIN ssdsdba.DataContainer ON ssdsdba.DataProducer.id = ssdsdba.DataContainer.DataProducerID_FK WHERE (ssdsdba.DataProducer.ParentID_FK = 31548) {noformat} where you use the DataProducerID for the new mooring deployment in the WHERE clause. ==> Change all of the 'File's in the dataContainerType' field to 'Stream's. h2. C. Procedures to be done after the mooring turn h4. Set up download info deployment and reporting Though not a real instrument, we configure a virtual 'dlinfo' instrument for the download scripts to attach download statistics data. We re-use the same device IDs for the M1 and M2 moorings, so it's best to configure this after the new mooring is out and all of those deployments have been closed. As the data are delivered "out of band" from the typical OASIS instruments we need to create a deployment for dlinfo instrument by hand. # The most direct way is to use the createDuplicateDeepDeployment service call. For example, to duplicate the 2009 M2 dlinfo deployment for the 2010 M2 deployment this call was executed: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer\|id=34496&p2Type=Date&p2Value=2010-04-03T22:00:00Z&p3Type=boolean&p3Value=false&p4Type=Date&p4Value=2010-04-03T23:00:00Z&p5Type=String&p5Value=getM2-download&p6Type=String&p6Value=&delimiter=\|. Of course you will need to adjust the DataProducer ID Date values for the new deployment you are creating. Here is the Key to the parameters: {noformat} Key: ---- http://localhost:8080/servlet/MetadataAccessServlet ?responseType=text &delimiter=| &objectToInvokeOn=DataProducerAccess &method=createDuplicateDeepDeployment &p1Type=DataProducer &p1Value=DataProducer|id=XXXX (XXXX is the ID of the deployment to copy) &p2Type=Date &p2Value=XXXXXX (XXXXXX is the start date of the new copy in XML format YYYY-MM-DDTHH:MM:SSZ) &p3Type=boolean &p3Value=(true|false) (this is to indicate if you want the original deployment to be closed) &p4Type=Date &p4Value=XXXXXX (XXXXXX is the end date for the original deployment (if p3Value is true)) &p5Type=String &p5Value=XXXXXX (XXXXXX is the DataProducer name of the new DataProducer) &p6Type=String &p6Value=XXXXXX (XXXXXX is the base URL to use for the new DataContainers that will be created). &delimiter=| {noformat}\\ Executing the createDuplicateDeepDeployment service call will return an ID for the new deployment. # Edit the new deployment record in the DataProducer table to adjust it's parentID_FK to be for the new mooring. While there edit times and name as appropriate. # Edit the getM? script on tsunami to use the proper device and parent ID (this will be the device ID of the torroid of the mooring deployment) and make sure that the '/oasis/bin/ssdsSubmit.pl $deviceId $parentId "$starttime_es,$endtime_es,$filesize,$rtnsts"' line in the script is configured to run. h4. Enable processing for other "virtual" devices # Turn on ClockSync processing. We also use a virtual device ID for these data. Simply uncomment the line for it in the ssds.cfg file. # Turn on ISUS processing. Simply uncomment the line for it in the ssds.cfg file. (Device ID re-used from previous deployment.) h4. Cycle links to previous deployment # Edit previous.html file in /mbari/ssdsdata/deployments to add a line for the new deployment and add the end date and archive url for the just closed deployment, e.g.: {noformat}
  • 200904 2009-04-29 to 2010-04-03
  • 201004 2010-04-03 to present
    • {noformat}\\ Mike McCann (First edit: 30 October 2007, Last updated: 1 May 2009)]]>     11796687     33161223 or _yyyy_.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the _yyyy_ directory. # Edit the mooring .cfg file and change the instrument deviceIDs and the path to the xml files to the newly deployed deviceIDs (aka ISI_IDs) and new xml directory. Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: {noformat} cd /cfg vi ssds.cfg {noformat} Refer to [Bob's documentation|http://mww.mbari.org/oasis/InternalDoc/OASIStoSSDS/doc/html/] for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata. # Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS _puckxml_ module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: {noformat} {noformat} In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, [http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd], will help with editing using a tool such as Oxygen or JEdit. Copy the files into the _yyyy_/xml subdirectory. +Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file;+ *{+}if you include other attributes they will overwrite what is in the database{+}*. # Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml file with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes. \[Note: With the M1 - October 2009 deployment we added the individual inductive modem microcats as children of the mooring and also configured a parallel deployment of the TString that produced the same data. With the M2 - April 2010 deployment we configured the individual IM mirocrocats and did not configure a parallel TString deployment. This is a better model for the system and is much easier to configure and produces more easily consumed data by downstream processes such as combineTS.pl.\] With future deployments we will not configure TString. # Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: {noformat} # The following is for ingesting OASIS data into SSDS # Please leave it alone; 'extract' et al will ignore it # 30 Oct 2007 Mike McCann ssds /ssdsdata/mooring/m1/2007/cfg/ssds.cfg {noformat} # Make sure that the \___ProcessRun lines in ssds.cfg refer to actual files in the xml directory. # Check the \___ProcessRun.xml files for correct DataFile and Resource uri/urls. # Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: {noformat} set SEND_SSDS $TRUE # Send the data to SSDS # Added 20 Jan 2005 Bob Herlien # if ( $SEND_SSDS == $TRUE ) then echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE" remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE & endif {noformat} # Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the [instrument swap procedure|SSDS:Instrument Swap on Oasis Mooring] and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer\|id=) on the wrong instrument deployment. # Test by: ## Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "\[SSDS:main\] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg. ## Check for metadata ingest in [Explorer|http://new-ssds.mbari.org:8080/ssds/faces/explorer.jsp] by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4. ## To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table. # Set up netCDF creation and plotting: ## Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: {noformat} # # Test M1 - Combine scripts look in Deployment Directory m1/200710 # ##set DDIR=200710 ##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \ -outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log {noformat} set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /mbari/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - " in SSDS, but that the argument to the \-mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...". ## Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page [http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html]). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate. ## Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products. h2. B. Procedure for doing the actual production mooring turn h4. Close existing mooring deployment # So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 27122) {noformat} (Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are to the actual end date. \\ For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the endDates so that everything on the recovered mooring is closed (or...as below \-rschramm 4/2010) \\ \\ {noformat} declare @myID as bigint set @myID = 31548 SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE id = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL UNION SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK FROM [ssdsdba].[DataProducer] WHERE parentid_fk in ( SELECT id FROM [ssdsdba].[DataProducer] WHERE parentid_fk = @myID and endDate is NULL) ORDER BY id {noformat} h4. Configure new mooring deployment # To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the +name+, +startDate+, and +nominalLatitude+ and +nominalLongitude+ attributes to reflect the production deployment. E.g.: {noformat}   Data for this mooring's watch circle are embedded in the uriString of this Resource. An application may parse for these parameters to use as criteria for issuing a warning if the GPS position is geater than warningDist (in km) from centerLat and centerLon (in decimal degrees WGS84) {noformat} The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSDS ingest will recognize them as new. # +Do not close (set endDates) the existing Test deployment until after the oasisToSSDS has run with the new platform deployment name+ and all new instrument deployments have been created in SSDS_Metadata. # See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadata, e.g.: {noformat} SELECT id, name, startDate, endDate, ParentID_FK FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') ORDER BY id DESC {noformat} Things to check for: ## All ParentIDs should not be null except for the Platform deployment. ## Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem ## All other instruments should be children of the platform deployment # Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.DataProducer.id >= 28873) ORDER BY ssdsdba.DataProducer.id DESC {noformat} You will need to set the DataProducer.id criteria to be appropriate for this deployment. # Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: {noformat} SELECT ssdsdba.DataProducerAssocDataProducerGroup.* FROM ssdsdba.DataProducerAssocDataProducerGroup WHERE (DataProducerGroupID_FK = 111) {noformat} and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group. # You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: {noformat} SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments' {noformat} Make sure you use the correct _M_ Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup. # If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: {noformat} SELECT ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name, ssdsdba.Device.id AS DeviceID, ssdsdba.Device.name AS DeviceName FROM ssdsdba.DataProducer INNER JOIN ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id WHERE (ssdsdba.Device.id = 1468) ORDER BY ssdsdba.DataProducer.id DESC {noformat} If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS \-c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami. # Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file. # You will also need to edit the DStoNetCDF.pl script to set the new name of the M1 deployment, e.g.: {noformat} # # SSDS mooring name Deployment lookup # my %ssdsMooringDeplNames = ( M0 => 'M0', M1 => 'M1 - October 2009', M1Test => 'Test M1 - October 2009', M2 => 'M2 - April 2009', {noformat} # Monitor the SSDS processing web page (e.g. [http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html]) for successful data processing. # When the mooring log message is sent to oasis with the actual time of deployment enter that as the startDate in the database.  All times (DTGs) in the SSDS_Metadata database are GMT. Make sure to set the startDate for the imctd microcat sensors too. This can be done with a query like this where you use the proper dataProduceIDs for the mooring and the imcd: {noformat} SELECT id, name, ParentID_FK, startDate, endDate, nominalDepth FROM ssdsdba.DataProducer WHERE (dataProducerType = 'Deployment') AND (ParentID_FK = 31564) OR (ParentID_FK = 31548) ORDER BY id DESC {noformat} # Because of some bug in SSDS ingest the dataContainerTypes of the outputs from the instrument deployments do not get properly assigned the values of 'Stream'.  This needs to be fixed so that the NDBC datatransfers will work. To fix it edit the SSDS_Metadata database starting with a query like this: {noformat} SELECT ssdsdba.DataContainer.* FROM ssdsdba.DataProducer INNER JOIN ssdsdba.DataContainer ON ssdsdba.DataProducer.id = ssdsdba.DataContainer.DataProducerID_FK WHERE (ssdsdba.DataProducer.ParentID_FK = 31548) {noformat} where you use the DataProducerID for the new mooring deployment in the WHERE clause. ==> Change all of the 'File's in the dataContainerType' field to 'Stream's. h2. C. Procedures to be done after the mooring turn h4. Set up download info deployment and reporting Though not a real instrument, we configure a virtual 'dlinfo' instrument for the download scripts to attach download statistics data. We re-use the same device IDs for the M1 and M2 moorings, so it's best to configure this after the new mooring is out and all of those deployments have been closed. As the data are delivered "out of band" from the typical OASIS instruments we need to create a deployment for dlinfo instrument by hand. # The most direct way is to use the createDuplicateDeepDeployment service call. For example, to duplicate the 2009 M2 dlinfo deployment for the 2010 M2 deployment this call was executed: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer\|id=*34496*&p2Type=Date&p2Value=*2010-04-03T22:00:00Z*&p3Type=boolean&p3Value=false&p4Type=Date&p4Value=*2010-04-03T23:00:00Z*&p5Type=String&p5Value=getM2-download&p6Type=String&p6Value=&delimiter=\|. Of course you will need to adjust the DataProducer ID Date values for the new deployment you are creating. Here is the Key to the parameters: {noformat} Key: ---- http://localhost:8080/servlet/MetadataAccessServlet ?responseType=text &delimiter=| &objectToInvokeOn=DataProducerAccess &method=createDuplicateDeepDeployment &p1Type=DataProducer &p1Value=DataProducer|id=XXXX (XXXX is the ID of the deployment to copy) &p2Type=Date &p2Value=XXXXXX (XXXXXX is the start date of the new copy in XML format YYYY-MM-DDTHH:MM:SSZ) &p3Type=boolean &p3Value=(true|false) (this is to indicate if you want the original deployment to be closed) &p4Type=Date &p4Value=XXXXXX (XXXXXX is the end date for the original deployment (if p3Value is true)) &p5Type=String &p5Value=XXXXXX (XXXXXX is the DataProducer name of the new DataProducer) &p6Type=String &p6Value=XXXXXX (XXXXXX is the base URL to use for the new DataContainers that will be created). &delimiter=| {noformat}\\ Executing the createDuplicateDeepDeployment service call will return an ID for the new deployment. # Edit the new deployment record in the DataProducer table to adjust it's parentID_FK to be for the new mooring. While there edit times and name as appropriate. # Edit the getM? script on tsunami to use the proper device and parent ID (this will be the device ID of the torroid of the mooring deployment) and make sure that the '/oasis/bin/ssdsSubmit.pl $deviceId $parentId "$starttime_es,$endtime_es,$filesize,$rtnsts"' line in the script is configured to run. Here's another example for the October 2010 M1 turn: {noformat} http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&\ p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer|id=35591&p2Type=Date&p2Value=2010-10-27T21:00:00Z&p3Type=boolean&p3Value=false&p4Type=Date&\ p4Value=2010-q0-27T20:30:00Z&p5Type=String&p5Value=getM1-download&p6Type=String&p6Value=&delimiter=| (This returned a new DataProducerID of 41556. The ParentID_FK for this record in the DataProducer table was then changed from 35388 to 41197 and the endDate cleared to null. Also, with the change to alternating deviceIDs implemented in 2010 the deviceID_FK was changed from 1698 to 1768 for this M1 deployment. {noformat} N.B. A rotating scheme for the virtual device IDs was implemented in 2010. This permits the steps described above to be executed prior to the mooring turn during the extensive dock-side test period. h4. Enable processing for other "virtual" devices # Turn on ClockSync processing. We also use a virtual device ID for these data. Simply uncomment the line for it in the ssds.cfg file. # Turn on ISUS processing. Simply uncomment the line for it in the ssds.cfg file. (Device ID re-used from previous deployment.) h4. Make all of the child instrument deployment start times the same as the mooring start time When SSDS receives a packet from an instrument that is not currently deployed it will create a Deployment record (in the DataProducer table) with a startDate that is set to the time of the first record received. As the mooring starts up all of the instrument deployments will have different start times based on when they each first saw data. This can present problems for the data processing that follows, especially for the jobs that aggregate the microcat data into a single ZT file that is used to produce the contour temperature and salinity wind stick plots. To prevent these problems it's best to edit the startDates of the child instrument deployments so that they are all the same. This is currently most easily done through Enterprise Manager with a query like below (this is for the 201010 M1 deployment) to get all the child instrument deployments: {noformat} SELECT * FROM ssdsdba.DataProducer WHERE (ParentID_FK = 41197) ORDER BY startDate {noformat} Then copy and paste the datetime string from one cell to the next. The times are GMT. h4. Cycle links to previous deployment # Edit previous.html file in /mbari/ssdsdata/deployments to add a line for the new deployment and add the end date and archive url for the just closed deployment, e.g.: {noformat}
  • 200904 2009-04-29 to 2010-04-03
  • 201004 2010-04-03 to present
    • {noformat}\\ # Add lines to DEPLOYMENTS file in dev/DPforSSDS/cimt/ for the just closed deployment, e.g.: {noformat} DStoNetCDF.pl -mooring M2 -deployment "M2 - April 2009" -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org -outputDir \ /mbari/ssdsdata/deployments -procClosed -verbose combineM.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments combineTS.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments combineAll.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments and /bin/cp /mbari/ssdsdata/deployments/m2/200904/OS_*.nc /mbari/FTP/pub/OceanSITES {noformat}\\ # Execute these lines to generate "closed deployment" data products and web pages and for submission to the OceanSITES GDAC # Confirm that the processing executed properly. Sometimes mangled timestamp data weasels its way into the data stream causing instrument netcdf files to be named with bogus start dates. (This can also happen during a deployment and is one of the maintenance tasks one should follow to keep the data flowing to where it needs to go.) The combine__.pl scripts will then create the OceanSITES formatted files encompassing the dates of all the instrument netcdf files. Incorrect instrument file names will create incorrect OS_MBARI* file names causing CenCOOS and NDBC/Ifremer to complain about the correctly named file not being updated with new data. The fix involves a purging of the bad files, double checking the metadata in SSDS and reprocessing, with perhaps additional checks in the instrument processing perl code to skip over bad records. This is best done at the Unix command line by cd'ing to the deployment directory, e.g. {noformat} cd /mbari/deployments/m1/201010 ls -lrt {noformat} and removing files that have bogus dates that do not represent the deployment start date. You should also remove all the directories and files in the gifs/ subdirectory. These will all get recreated when you run DStoNetCDF.pl and the combine__.pl scripts. With luck, simply removing the bad files and rerunning the processing scripts will fix things. If not, identify where the bad dates are coming from and fix as appropriate. Mike McCann (First edit: 30 October 2007, Last updated: 1 March 2012)]]>     33095687     19071253 19038487 2 2006-10-17 21:48:40.373 2026-02-04 08:33:05.020 1867869 1835105 6422536 2008-10-09 22:18:04.253 2026-02-04 08:33:05.048 18645302 18579787 3637814 6 false 0.0 0 0 17858815 5 false 0.0 0 0 3114486 5 false 0.0 0 0 3637814 5 false 0.0 0 0 2162834 6 false 0.0 0 0 1835140 6 false 0.0 0 0 18579795 6 false 0.0 0 0 16777265 5 false 0.0 0 0 3114486 6 false 0.0 0 0 17858815 6 false 0.0 0 0 1835091 6 false 0.0 0 0 1835090 6 false 0.0 0 0 1835091 5 false 0.0 0 0 2162834 5 false 0.0 0 0 17858807 5 false 0.0 0 0 1835090 5 false 0.0 0 0 18579795 5 false 0.0 0 0 17858807 6 false 0.0 0 0 16777265 6 false 0.0 0 0 1835140 5 false 0.0 0 0