file date stamp as a variable

[Maurice Vergeer]

dear list members

I have about 1400 small text files that need to be converted to one
single system file (sav). I found a way to do this on the
spsstools.net site. However, I also want to include the time stamp of
these text files (time and date the file was created on disc) as a
variable in the system file. Is there a way to do this in an automated
way?

thanks
Maurice Vergeer

--
___________________________________________________________________
Maurice Vergeer
Department of communication
Radboud University  (www.ru.nl)
PO Box 9104
NL-6500 HE Nijmegen
The Netherlands

Visiting Professor Yeungnam University, Gyeongsan, South Korea

contact:
E: [hidden email]
T: [phone] (direct)/ 3612372 (secretary) / maurice.vergeer (skype)
personal webpage: www.mauricevergeer.nl
blog:  http://blog.mauricevergeer.nl/
Journalism: www.journalisteninhetdigitaletijdperk.nl
CENMEP New Media and European Parliament Elections 2009
http://mauricevergeer.ruhosting.nl/cenmep

Recent publications:
- Vergeer, M. & Pelzer, B. (2009). Consequences of media and Internet
use for offline and online network capital and well-being. A causal
model approach. Journal of Computer-Mediated Communication, 15,
189-210.
-Vergeer, M., Coenders, M. & Scheepers, P. (2009). Time spent on
television in European countries. In R.P. Konig, P.W.M. Nelissen, &
F.J.M. Huysmans (Eds.), Meaningful media: Communication Research on
the Social Construction of Reality (54-73). Nijmegen, The Netherlands:
Tandem Felix.
- Hermans, L., Vergeer, M., &  d’Haenens, L. (2009). Internet in the
daily life of journalists. Explaining the use of the Internet through
work-related characteristics and professional opinions. Journal of
Computer-Mediated Communication, 15, 138-157.
- Hermans, L., Vergeer, M., & Pleijter, A. (2009). Internet Adoption
in the Newsroom: Journalists’ use of the Internet explained by
Attitudes and Perceived Functions. Communications. The European
Journal of Communication, 34(1), 55-71.
- Vergeer, M., & Hermans, L. (2008). Analyzing online political
discussions: Methodological considerations. Javnost/The Public, 15(2),
37-56.
___________________________________________________________________

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Albert-Jan Roskam]

Hi Maurice,

That's quite easy with Python. The code below should do what you want, even though it's untested. I'm pretty sure that it works, but you need to give the command that you use to read the txt file (do they have a separator? are they fixed width files?). And of course, you need to set the correct dir (remember to prefix it with r"  if you use backslashes.)

BEGIN PROGRAM.
import os, time, glob, spss

txts = glob.glob("d:/temp/*.txt")
for cnt, txt in enumerate(txts):
  mode, ino, dev, nlink, uid, gid, size, atime, mtime, ctime = os.stat(txt)
  stamp = time.strftime("%Y-%m-%d", time.localtime(mtime)) # last modification date of the file
  spss.Submit("get file = '%s'." % txt) # --- replace this with your GET DATA command, use triple quotes if needed ---.
  spss.Submit("string moddate (a10).\ncompute moddate = %s." % stamp)
  if cnt == 0:
   spss.Submit("save outfile = 'd:/temp/merged.sav'.")   
  elif cnt % 50 == 0:
    pct = float((cnt+1)) / len(txts)
    spss.Submit("echo '** Processing file %s of %s (%4.2f%%)'." % (cnt+1, len(txts), pct)
    spss.Submit("add files / file = 'd:/temp/merged.sav' / file = *.")
    spss.Submit("save outfile = 'd:/temp/merged.sav'.")
  else:
    spss.Submit("add files / file = 'd:/temp/merged.sav' / file = *.")
    spss.Submit("xsave outfile = 'd:/temp/merged.sav'.") # use XSAVE here
spss.Submit("sort cases by moddate.")
spss.Submit("save outfile = 'd:/temp/merged.sav'.")
END PROGRAM.

Cheers!!
Albert-Jan

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In the face of ambiguity, refuse the temptation to guess.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

--- On Sat, 2/20/10, Maurice Vergeer <[hidden email]> wrote:

From: Maurice Vergeer <[hidden email]>
Subject: [SPSSX-L] file date stamp as a variable
To: [hidden email]
Date: Saturday, February 20, 2010, 10:35 AM

dear list members

I have about 1400 small text files that need to be converted to one
single system file (sav). I found a way to do this on the
spsstools.net site. However, I also want to include the time stamp of
these text files (time and date the file was created on disc) as a
variable in the system file. Is there a way to do this in an automated
way?

thanks
Maurice Vergeer

--
___________________________________________________________________
Maurice Vergeer
Department of communication
Radboud University  (www.ru.nl)
PO Box 9104
NL-6500 HE Nijmegen
The Netherlands

Visiting Professor Yeungnam University, Gyeongsan, South Korea

contact:
E: m.vergeer@...
T: [phone] (direct)/ 3612372 (secretary) / maurice.vergeer (skype)
personal webpage: www.mauricevergeer.nl
blog:  http://blog.mauricevergeer.nl/
Journalism: www.journalisteninhetdigitaletijdperk.nl
CENMEP New Media and European Parliament Elections 2009
http://mauricevergeer.ruhosting.nl/cenmep

Recent publications:
- Vergeer, M. & Pelzer, B. (2009). Consequences of media and Internet
use for offline and online network capital and well-being. A causal
model approach. Journal of Computer-Mediated Communication, 15,
189-210.
-Vergeer, M., Coenders, M. & Scheepers, P. (2009). Time spent on
television in European countries. In R.P. Konig, P.W.M. Nelissen, &
F.J.M. Huysmans (Eds.), Meaningful media: Communication Research on
the Social Construction of Reality (54-73). Nijmegen, The Netherlands:
Tandem Felix.
- Hermans, L., Vergeer, M., &  d’Haenens, L. (2009). Internet in the
daily life of journalists. Explaining the use of the Internet through
work-related characteristics and professional opinions. Journal of
Computer-Mediated Communication, 15, 138-157.
- Hermans, L., Vergeer, M., & Pleijter, A. (2009). Internet Adoption
in the Newsroom: Journalists’ use of the Internet explained by
Attitudes and Perceived Functions. Communications. The European
Journal of Communication, 34(1), 55-71.
- Vergeer, M., & Hermans, L. (2008). Analyzing online political
discussions: Methodological considerations. Javnost/The Public, 15(2),
37-56.
___________________________________________________________________

=====================
To manage your subscription to SPSSX-L, send a message to
LISTSERV@... (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Ruben Geert van den Berg]

Dear Maurice,

Exactly which syntax from SPSStools.net are you using? Does your .sav file include filenames? If yes, then you could merge the creation date and time as well as the file sizes into your .sav file with a match files command from an external table including this information. But where do we get such a table? I thought SYSFILE INFO within OMS would do the trick here but it seems to work only on .sav files, not .txt. However, the HOST command below seems to work (for me at least). I pasted the GET DATA command from the GUI with tabs and spaces as delimiters. Well, maybe it's not as nice as the Python solution but I hope you'll find it useful.

Best regards,

Ruben van den Berg

Methodologist

TNS NIPO

E: [hidden email]

P: [phone]

I: www.tns-nipo.com

cd 'c:\temp'.

* OMS.

OMS

 /SELECT ALL

 /IF COMMANDS = ["Host"]

 /DESTINATION FORMAT = TABTEXT

  OUTFILE = "textfiles.txt"

 /TAG = "textfiles".

host command=['dir c:\temp\*.txt'].

omsend tag=['textfiles'].

GET DATA  /TYPE = TXT

 /FILE = 'C:\Temp\textfiles.txt'

 /DELCASE = LINE

 /DELIMITERS = "\t "

 /ARRANGEMENT = DELIMITED

 /FIRSTCASE = 1

 /IMPORTCASE = ALL

 /VARIABLES =

 V1 A10

 V2 A12

 V3 A14

 V4 A20

 V5 A17

 V6 A5

 V7 A4

 V8 A5 .

CACHE.

EXECUTE.

DATASET NAME D1 WINDOW=FRONT.

sel if ind(v2,":")>0.

exe.

> Date: Sat, 20 Feb 2010 10:35:41 +0100
> From: [hidden email]
> Subject: file date stamp as a variable
> To: [hidden email]
>
> dear list members
>
> I have about 1400 small text files that need to be converted to one
> single system file (sav). I found a way to do this on the
> spsstools.net site. However, I also want to include the time stamp of
> these text files (time and date the file was created on disc) as a
> variable in the system file. Is there a way to do this in an automated
> way?
>
> thanks
> Maurice Vergeer
>
> --
> ___________________________________________________________________
> Maurice Vergeer
> Department of communication
> Radboud University (www.ru.nl)
> PO Box 9104
> NL-6500 HE Nijmegen
> The Netherlands
>
> Visiting Professor Yeungnam University, Gyeongsan, South Korea
>
> contact:
> E: [hidden email]
> T: [phone] (direct)/ 3612372 (secretary) / maurice.vergeer (skype)
> personal webpage: www.mauricevergeer.nl
> blog: http://blog.mauricevergeer.nl/
> Journalism: www.journalisteninhetdigitaletijdperk.nl
> CENMEP New Media and European Parliament Elections 2009
> http://mauricevergeer.ruhosting.nl/cenmep
>
> Recent publications:
> - Vergeer, M. & Pelzer, B. (2009). Consequences of media and Internet
> use for offline and online network capital and well-being. A causal
> model approach. Journal of Computer-Mediated Communication, 15,
> 189-210.
> -Vergeer, M., Coenders, M. & Scheepers, P. (2009). Time spent on
> television in European countries. In R.P. Konig, P.W.M. Nelissen, &
> F.J.M. Huysmans (Eds.), Meaningful media: Communication Research on
> the Social Construction of Reality (54-73). Nijmegen, The Netherlands:
> Tandem Felix.
> - Hermans, L., Vergeer, M., & d’Haenens, L. (2009). Internet in the
> daily life of journalists. Explaining the use of the Internet through
> work-related characteristics and professional opinions. Journal of
> Computer-Mediated Communication, 15, 138-157.
> - Hermans, L., Vergeer, M., & Pleijter, A. (2009). Internet Adoption
> in the Newsroom: Journalists’ use of the Internet explained by
> Attitudes and Perceived Functions. Communications. The European
> Journal of Communication, 34(1), 55-71.
> - Vergeer, M., & Hermans, L. (2008). Analyzing online political
> discussions: Methodological considerations. Javnost/The Public, 15(2),
> 37-56.
> ___________________________________________________________________
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD

---

New Windows 7: Find the right PC for you. Learn more.

[Albert-Jan Roskam]

Hi,

 

I just compared the code that I previously posted to an existing syntax, and xsave appears not to work (I made the same reasoning error twice now!). The code below is tested & working.

 

* sample data.
get file = 'c:/program files/spss/employee data.sav'.
select if gender eq 'f'.
save translate outfile='d:\temp\women.txt'  /type=tab /map /replace /fieldnames.
get file = 'c:/program files/spss/employee data.sav'.
select if gender eq 'm'.
save translate outfile='d:\temp\men.txt'  /type=tab /map /replace /fieldnames.

* actual program.
* set mprint = on.

BEGIN PROGRAM.
import os, time, glob, spss

cmd = r"""
get data  /type = txt
 /file = '%s'
 /delcase = line
 /delimiters = "\t"
 /arrangement = delimited
 /firstcase = 2
 /importcase = all
 /variables =
 id f2.1
 gender a1
 bdate a10
 educ f2.1
 jobcat f1.0
 salary f6.2
 salbegin f5.2
 jobtime f2.1
 prevexp f3.2
 minority f1.0
 .
cache.
execute.
"""

txts = glob.glob("d:/temp/*.txt") # make sure there are no non-data txt files!
for cnt, txt in enumerate(txts):
  mode, ino, dev, nlink, uid, gid, size, atime, mtime, ctime = os.stat(txt)
  stamp = time.strftime("%Y-%m-%d", time.localtime(mtime)) # last modification date of the file
  spss.Submit(cmd % txt)
  spss.Submit("string moddate (a10).\ncompute moddate = '%s'." % stamp)
  pct = float((cnt+1)) / len(txts) * 100
  spss.Submit("echo '** Processing file %s of %s (%4.2f%%)'." % (cnt+1, len(txts), pct))
  if cnt == 0:
   spss.Submit("save outfile = 'd:/temp/merged.sav'.")   
  else:
    spss.Submit("add files / file = 'd:/temp/merged.sav' / file = *.")
spss.Submit("sort cases by moddate.")
spss.Submit("save outfile = 'd:/temp/merged.sav'.")
END PROGRAM.


Cheers!!
Albert-Jan

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In the face of ambiguity, refuse the temptation to guess.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

--- On Sat, 2/20/10, Maurice Vergeer <[hidden email]> wrote:

From: Maurice Vergeer <[hidden email]>
Subject: [SPSSX-L] file date stamp as a variable
To: [hidden email]
Date: Saturday, February 20, 2010, 10:35 AM

dear list members

I have about 1400 small text files that need to be converted to one
single system file (sav). I found a way to do this on the
spsstools.net site. However, I also want to include the time stamp of
these text files (time and date the file was created on disc) as a
variable in the system file. Is there a way to do this in an automated
way?

thanks
Maurice Vergeer

--
___________________________________________________________________
Maurice Vergeer
Department of communication
Radboud University  (www.ru.nl)
PO Box 9104
NL-6500 HE Nijmegen
The Netherlands

Visiting Professor Yeungnam University, Gyeongsan, South Korea

contact:
E: m.vergeer@...
T: [phone] (direct)/ 3612372 (secretary) / maurice.vergeer (skype)
personal webpage: www.mauricevergeer.nl
blog:  http://blog.mauricevergeer.nl/
Journalism: www.journalisteninhetdigitaletijdperk.nl
CENMEP New Media and European Parliament Elections 2009
http://mauricevergeer.ruhosting.nl/cenmep

Recent publications:
- Vergeer, M. & Pelzer, B. (2009). Consequences of media and Internet
use for offline and online network capital and well-being. A causal
model approach. Journal of Computer-Mediated Communication, 15,
189-210.
-Vergeer, M., Coenders, M. & Scheepers, P. (2009). Time spent on
television in European countries. In R.P. Konig, P.W.M. Nelissen, &
F.J.M. Huysmans (Eds.), Meaningful media: Communication Research on
the Social Construction of Reality (54-73). Nijmegen, The Netherlands:
Tandem Felix.
- Hermans, L., Vergeer, M., &  d’Haenens, L. (2009). Internet in the
daily life of journalists. Explaining the use of the Internet through
work-related characteristics and professional opinions. Journal of
Computer-Mediated Communication, 15, 138-157.
- Hermans, L., Vergeer, M., & Pleijter, A. (2009). Internet Adoption
in the Newsroom: Journalists’ use of the Internet explained by
Attitudes and Perceived Functions. Communications. The European
Journal of Communication, 34(1), 55-71.
- Vergeer, M., & Hermans, L. (2008). Analyzing online political
discussions: Methodological considerations. Javnost/The Public, 15(2),
37-56.
___________________________________________________________________

=====================
To manage your subscription to SPSSX-L, send a message to
LISTSERV@... (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Maurice Vergeer]

Dear Albert-Jan and Ruben,

I appreciate your solutions to my problem very much. Today I will test these and let you know what my findings are.

Best wishes
Maurice

 

[Maurice Vergeer]

Hi Albert-Jan,

After downgrading to python 2.5 and installing the plugin, I'm working with your spss/python-code.
I run into one small but crucial problem: the final merged file only contains the last textfile instead of all textfiles.
there may be two solutions:
-rewriting the code so the final merged file is correct.
-rewriting the code that individual sav-files are saved. With another spss-code I could merged them to one file.

could you help me out with this?

By the way, judging from your name you are Dutch?


Best wishes
Maurice

2010/2/22 Albert-Jan Roskam <[hidden email]>

Hi,

 

I just compared the code that I previously posted to an existing syntax, and xsave appears not to work (I made the same reasoning error twice now!). The code below is tested & working.

 

* sample data.
get file = 'c:/program files/spss/employee data.sav'.
select if gender eq 'f'.
save translate outfile='d:\temp\women.txt'  /type=tab /map /replace /fieldnames.
get file = 'c:/program files/spss/employee data.sav'.
select if gender eq 'm'.
save translate outfile='d:\temp\men.txt'  /type=tab /map /replace /fieldnames.

* actual program.
* set mprint = on.

BEGIN PROGRAM.
import os, time, glob, spss

cmd = r"""
get data  /type = txt
 /file = '%s'
 /delcase = line
 /delimiters = "\t"
 /arrangement = delimited
 /firstcase = 2
 /importcase = all
 /variables =
 id f2.1
 gender a1
 bdate a10
 educ f2.1
 jobcat f1.0
 salary f6.2
 salbegin f5.2
 jobtime f2.1
 prevexp f3.2
 minority f1.0
 .
cache.
execute.
"""

txts = glob.glob("d:/temp/*.txt") # make sure there are no non-data txt files!

for cnt, txt in enumerate(txts):
  mode, ino, dev, nlink, uid, gid, size, atime, mtime, ctime = os.stat(txt)
  stamp = time.strftime("%Y-%m-%d", time.localtime(mtime)) # last modification date of the file

  spss.Submit(cmd % txt)

  spss.Submit("string moddate (a10).\ncompute moddate = '%s'." % stamp)

  pct = float((cnt+1)) / len(txts) * 100
  spss.Submit("echo '** Processing file %s of %s (%4.2f%%)'." % (cnt+1, len(txts), pct))

  if cnt == 0:
   spss.Submit("save outfile = 'd:/temp/merged.sav'.")   

  else:
    spss.Submit("add files / file = 'd:/temp/merged.sav' / file = *.")

spss.Submit("sort cases by moddate.")
spss.Submit("save outfile = 'd:/temp/merged.sav'.")
END PROGRAM.


Cheers!!
Albert-Jan

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In the face of ambiguity, refuse the temptation to guess.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

--- On Sat, 2/20/10, Maurice Vergeer <[hidden email]> wrote:

From: Maurice Vergeer <[hidden email]>
Subject: [SPSSX-L] file date stamp as a variable

To: [hidden email]

Date: Saturday, February 20, 2010, 10:35 AM

dear list members

I have about 1400 small text files that need to be converted to one
single system file (sav). I found a way to do this on the
spsstools.net site. However, I also want to include the time stamp of
these text files (time and date the file was created on disc) as a
variable in the system file. Is there a way to do this in an automated
way?

thanks
Maurice Vergeer

--
___________________________________________________________________
Maurice Vergeer
Department of communication
Radboud University  (www.ru.nl)
PO Box 9104
NL-6500 HE Nijmegen
The Netherlands

Visiting Professor Yeungnam University, Gyeongsan, South Korea

contact:
E: m.vergeer@...
T: [phone] (direct)/ 3612372 (secretary) / maurice.vergeer (skype)
personal webpage: www.mauricevergeer.nl
blog:  http://blog.mauricevergeer.nl/
Journalism: www.journalisteninhetdigitaletijdperk.nl
CENMEP New Media and European Parliament Elections 2009
http://mauricevergeer.ruhosting.nl/cenmep

Recent publications:
- Vergeer, M. & Pelzer, B. (2009). Consequences of media and Internet
use for offline and online network capital and well-being. A causal
model approach. Journal of Computer-Mediated Communication, 15,
189-210.
-Vergeer, M., Coenders, M. & Scheepers, P. (2009). Time spent on
television in European countries. In R.P. Konig, P.W.M. Nelissen, &
F.J.M. Huysmans (Eds.), Meaningful media: Communication Research on
the Social Construction of Reality (54-73). Nijmegen, The Netherlands:
Tandem Felix.
- Hermans, L., Vergeer, M., &  d’Haenens, L. (2009). Internet in the
daily life of journalists. Explaining the use of the Internet through
work-related characteristics and professional opinions. Journal of
Computer-Mediated Communication, 15, 138-157.
- Hermans, L., Vergeer, M., & Pleijter, A. (2009). Internet Adoption
in the Newsroom: Journalists’ use of the Internet explained by
Attitudes and Perceived Functions. Communications. The European
Journal of Communication, 34(1), 55-71.
- Vergeer, M., & Hermans, L. (2008). Analyzing online political
discussions: Methodological considerations. Javnost/The Public, 15(2),
37-56.
___________________________________________________________________

=====================
To manage your subscription to SPSSX-L, send a message to
LISTSERV@... (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

--
___________________________________________________________________
Maurice Vergeer
Department of communication
Radboud University  (www.ru.nl)
PO Box 9104
NL-6500 HE Nijmegen
The Netherlands

Visiting Professor Yeungnam University, Gyeongsan, South Korea

contact:
E: [hidden email]
T: [phone] (direct)/ 3612372 (secretary) / maurice.vergeer (skype)
personal webpage: www.mauricevergeer.nl
blog:  http://blog.mauricevergeer.nl/
Journalism: www.journalisteninhetdigitaletijdperk.nl
CENMEP New Media and European Parliament Elections 2009 http://mauricevergeer.ruhosting.nl/cenmep

Recent publications:
- Vergeer, M. & Pelzer, B. (2009). Consequences of media and Internet use for offline and online network capital and well-being. A causal model approach. Journal of Computer-Mediated Communication, 15, 189-210.
-Vergeer, M., Coenders, M. & Scheepers, P. (2009). Time spent on television in European countries. In R.P. Konig, P.W.M. Nelissen, & F.J.M. Huysmans (Eds.), Meaningful media: Communication Research on the Social Construction of Reality (54-73). Nijmegen, The Netherlands: Tandem Felix.
- Hermans, L., Vergeer, M., &  d’Haenens, L. (2009). Internet in the daily life of journalists. Explaining the use of the Internet through work-related characteristics and professional opinions. Journal of Computer-Mediated Communication, 15, 138-157.
- Hermans, L., Vergeer, M., & Pleijter, A. (2009). Internet Adoption in the Newsroom: Journalists’ use of the Internet explained by Attitudes and Perceived Functions. Communications. The European Journal of Communication, 34(1), 55-71.
- Vergeer, M., & Hermans, L. (2008). Analyzing online political discussions: Methodological considerations. Javnost/The Public, 15(2), 37-56.
___________________________________________________________________