BMI-219 – Scientific Software Development Frequently Asked Questions

Why isn't my CGI script working?

• Either your script is not being executed, or
• your script is generating an error.

Why isn't my CGI script being run?

• Your CGI directory must be located in a path that looks something like /home/socr/b/bmi219/current/z/conrad/cgi-bin
  • The directory must be exactly four levels below bmi219.
• Make sure the first line of your script looks like:

  #!/usr/bin/python2

• Make sure your script is executable.

  $ cd /home/socr/b/bmi219/current/z/conrad/cgi-bin
  $ ls -l
  total 4
  -rw-rw-r-- 1 conrad bmi219 179 Apr 17 09:58 example.cgi
  $ chmod +x example.cgi
  $ ls -l
  total 4
  -rwxrwxr-x 1 conrad bmi219 179 Apr 17 09:58 example.cgi
  $ ./example.cgi
  ???

  In the above example, -rw-rw-r-- indicates that the file is not executable. The chmod command adds execute permission for everyone.
• Check to see if Apache thinks your script is being invoked:

  $ cd /usr/local/www/logs/plato-test-httpd/bmi219
  $ grep example.cgi bmi219-ssl-access_log
  169.230.21.23 - - [17/Apr/2017:09:56:19 -0700] "GET /z/conrad/cgi-bin/example.cgi HTTP/1.1" 401 362 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0" 0/29321
  169.230.21.23 - conrad [17/Apr/2017:09:56:22 -0700] "GET /z/conrad/cgi-bin/example.cgi HTTP/1.1" 500 613 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0" 0/117099
  169.230.21.23 - conrad [17/Apr/2017:09:56:46 -0700] "GET /z/conrad/cgi-bin/example.cgi HTTP/1.1" 200 50 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0" 0/155331

  The HTTP return code appears immediately after the HTTP request.
  • 401 is "Unauthorized", and is normal when web site requires login.
  • 200 is "Success", so your script is being called
  • 500 is "Internal Server Error", so your script is being called but is generating output that Apache is treating as unacceptable HTTP response.

Why is my CGI script failing?

• Try running your script on the server

  $ cd /home/socr/b/bmi219/current/z/conrad/cgi-bin
  $ ./example.cgi
  ???

  Your script should run to completion. It may complain about bad or missing input, but it should not generate syntax errors.
• Make sure your script output starts with HTTP headers, and don't forget the blank line terminating the headers.
• Try going to the CGI URL directly in your browser and see if you get the response generated by your CGI script.
• Use "View Source" in the browser to see how far your script got.
• Check the logs to see what Apache is reporting

  $ cd /usr/local/www/logs/plato-test-httpd/bmi219
  $ grep example.cgi bmi219-ssl-error_log
  [Mon Apr 17 09:56:22.793106 2017] [cgi:error] [pid 34932] [client 169.230.21.23:39853] End of script output before headers: example.cgi
  [Mon Apr 17 09:58:34.034631 2017] [cgi:error] [pid 34932] [client 169.230.21.23:40281] AH01215:   File "/home/socr/b/bmi219/current/z/conrad/cgi-bin/example.cg ", line 7, in 
  [Mon Apr 17 10:25:06.924289 2017] [cgi:error] [pid 55993] [client 169.230.21.23:40643] malformed header from script 'example.cgi': Bad header: 
  [Mon Apr 17 10:25:06.925424 2017] [cgi:error] [pid 55993] [client 169.230.21.23:40643] AH01215:   File "/home/socr/b/bmi219/current/z/conrad/cgi-bin/example.cg ", line 7, in 
  

  If your script is throwing an exception, the (badly formatted) traceback will appear in the error log.

Can I see CGI script tracebacks without munging through log files?

• The cgitb module catches errors for you and tries to generate nicely formatted HTML output from the traceback. The problem is that if your script generated HTTP headers (e.g., Content-Type: text/xml) first and then throws an exception, the output from cgitb is treated as your type instead of HTML. To avoid this problem, you should buffer output from your script as a string, and only send the string if the script completes successfully.

  #!/usr/bin/python2
  
  # Enable CGI traceback generations.  Turn off for production
  import cgitb; cgitb.enable()
  # Save standard output and replace it with a string buffer.
  # All script output will be appended to the string buffer.
  import sys, StringIO
  orig_stdout = sys.stdout
  sys.stdout = StringIO.StringIO()
  
  print "Content-Type: text/xml"
  print
  print ""
  print ""
  # If an exception occurs in the script, script output
  # will not be sent back.  Instead, cgitb will generate
  # HTML version of the traceback.
  raise ValueError("bad value")
  print "This is content"
  print ""
  print ""
  
  # Script must have completed successfully so send
  # script output to the saved standard output.
  orig_stdout.write(sys.stdout.getvalue())

Can I use external Python packages in my CGI scripts?

• The easiest way is to put the Python packages/modules in the cgi-bin directory next to the CGI scripts. If you are installing using pip, use the -t option to specify where the package should be installed. For example, to install the social-auth package in my cgi-bin directory, use:

  $ pip install -t /home/socr/b/bmi219/current/z/conrad/cgi-bin social-auth

  Note that you need to repeat this for each team member as well as the team production site since the installed package is only available to the script in the same directory.

Why does my CGI script work on the server but fail when accessed from the web?

• CGI scripts have different permissions depending on how they are invoked. When run from the shell on the server, a script is running as the invoking user, e.g., conrad; when run as a CGI script, the same script is running as user apache. This difference in permission is often the cause of unexpected failures, such as inability to open sqlite3 databases.

Why does my CGI script work when I run it manually but fail when accessed from my web page?

• CGI scripts have different permissions depending on how they are invoked. When run from the shell on the server, a script is running as the invoking user, e.g., conrad; when run as a CGI script, the same script is running as user apache. This difference in permission is often the cause of unexpected failures, such as inability to open sqlite3 databases.

Why can't my CGI script open my database when accessed from my web page?

• sqlite3 wants write permission to the directory containing the database file when inserting or updating rows. Even if the database file itself is writable, your code can still get the dreaded sqlite3.OperationalError: unable to open database file error if the directory is not writable.

  $ ls -la
  total 16
  dr-xr-xr-x 2 conrad ferrin 3864 Apr 19 15:40 ./
  drwxrwxr-x 6 conrad ferrin 3864 Apr 19 15:39 ../
  -rw-r--r-- 1 conrad ferrin 2048 Apr 19 15:29 t.db
  -rw-rw-r-- 1 conrad ferrin  237 Apr 19 15:31 t.py
  $ cat t.py
  import sqlite3
  
  db = sqlite3.connect("t.db")
  c = db.cursor()
  #c.execute("create table t1 ( 'name' char(16) )");
  c.execute("insert into t1 values ('conrad')")
  db.commit()
  for row in c.execute("select * from t1"):
      print row
  db.close()
  $ python t.py
  Traceback (most recent call last):
    File "t.py", line 6, in 
      c.execute("insert into t1 values ('conrad')")
  sqlite3.OperationalError: unable to open database file

  This is a particularly subtle error if your script works when run on the server but fails when accessed from a web page because the same script is being run by two different users: user who logged in to the server and user apache.
• For BMI219, we created a data directory that is owned by user apache but writable by group bmi219. If your databases are created in there and made generally writable, then apache should be able to update your database.

  $ ls -la data
  total 16
  drwxrwxr-x 2 apache bmi219 3864 Apr 19 15:52 ./
  drwxrwsr-x 6 conrad bmi219 3864 Apr 19 15:49 ../
  -rw-rw-r-- 1 conrad ferrin 2048 Apr 19 15:52 t.db
  -rw-rw-r-- 1 conrad ferrin  237 Apr 19 15:50 t.py
  $ chmod go+w data/t.db
  $ ls -la data
  total 16
  drwxrwxr-x 2 apache bmi219 3864 Apr 19 15:52 ./
  drwxrwsr-x 6 conrad bmi219 3864 Apr 19 15:49 ../
  -rw-rw-rw- 1 conrad ferrin 2048 Apr 19 15:52 t.db
  -rw-rw-r-- 1 conrad ferrin  237 Apr 19 15:50 t.py

• Making databases generally writable is not secure. However, this is the path of least resistance for a short course. If you want to be more secure, you can write a CGI script to create the database and invoke it via a web page; this would make apache the owner of the database file and therefore updatable via the web without having to be made generally writable.

Why is my web page being reloaded even when I'm using AJAX?

• A common cause of this problem is using a button's onclick event handler to initiate an AJAX request. If the type attribute for an HTML button is not specified, it defaults to submit. If the onclick handler returns false, then the form is not submitted. If it returns true, then the form is submitted and the entire web page is replaced by the response to the form submsstion; any AJAX request initiated by the replaced page is silently discarded.
• The workarounds are:
  • Set the button type attribute to button.
  • Make sure the button onclick handler returns false. In specifying the Javascript event handler code, be sure to either return the value from whatever function is being called, or always return false after calling the function.
  • Use a different input tag to initiate AJAX requests, e.g., onchange event handler for a text tag.