HELP How to display a csv file to a graph (high charts) dynamically?

by: tonyphoang, 9 years ago

Last edited: 9 years ago

I'm currently on Python lesson #27.

I have a csv folder located in /var/www/FlaskApp/FlaskApp/csv/ that has multiple csv files. There is a script that generates these csv files.

I was hoping to have HTML push arguments to python to execute this script then display the resulting csv file as a graph using high charts.

How would I categorize and display each csv as a graph? Can someone help me?

Here are some things that I've been able to piece together

1. reading csv into a list

import csv
with open('experiment1/experiment1.csv', 'r') as f:
    reader = csv.reader(f)
    your_list = list(reader)

print your_list





list_csv_from_all_dir.py 

import glob
print glob.glob("/var/www/FlaskApp/FlaskApp/csv/*.csv")



2. content_management.py

def Content():
    sections = {"section1":[["results1","/url1"],
                            ["results2","/url2"],
                            ["results3","/url3"]],
				"section2 Dev":[]}

    return sections



my __init__.py

@app.route('/results/')
def results():
    try:
		pageType='results'
		sections = Content() #for file management
		return render_template("results.html",pageType=pageType,sections=sections)
    except Exception as e:
	    return render_template("500.html", error = str(e))






Also how do I POST arguments from HTML to python?
Would I just include a

methods=["GET","POST"]

into my decorator and call it  in my .html code

 action="" class="form-inline" method="post"



You must be logged in to post. Please login or register an account.



For the second question, I believe I found the answer but would like some verification from you if that's the right path.
http://stackoverflow.com/questions/11556958/sending-data-from-html-form-to-python-flask

-tonyphoang 9 years ago

You must be logged in to post. Please login or register an account.

What's the use of the form? I don't see any code that handles uploads, what's the form doing?

Your entire question is answered by various tutorial here: https://pythonprogramming.net/dashboard/#tab_webdev

For form input, look at the https://pythonprogramming.net/flask-get-post-requests-handling-tutorial/

For graphs, I've done that too: https://pythonprogramming.net/adding-js-plugins-flask-highcharts-example/

Otherwise, not sure what kind of arguments you're trying to push, but it's all there.

-Harrison 9 years ago

You must be logged in to post. Please login or register an account.


Thanks for the quick reply.

What I'm trying to do is pass arguments from HTML to python to control launch a script. When the script is completed, it will output a csv file.

I would like to have some type of management system that automatically categorizes the csv.

-tonyphoang 9 years ago

You must be logged in to post. Please login or register an account.


I believe I know how to get the POST working

my __init__.py

# Define a route for the default URL, which loads the form
@app.route('/')
def form():
    return render_template('form_submit.html')

# Define a route for the action of the form, for example '/hello/'
# We are also defining which type of requests this route is 
# accepting: POST requests in this case
@app.route('/hello/', methods=['POST'])
def hello():
    name=request.form['yourname']
    email=request.form['youremail']
    return render_template('form_action.html', name=name, email=email)


my form_submit.html

<html>
    <head>
        <title>Handle POST requests with Flask</title>
        <link rel=stylesheet type=text/css href="{{ url_for('static', filename='style.css') }}">
    </head>
    <body>
        <div id="container">
            <div class="title">
                <h1>POST request with Flask</h1>
            </div>
            <div id="content">
                <form method="post" action="{{ url_for('hello') }}">
                  <label for="yourname">Please enter your name:</label>
                  <input type="text" name="yourname" /><br />
                  <label for="youremail">Please enter your email:</label>
                  <input type="text" name="youremail" /><br />
                  <input type="submit" />
                </form>
            </div>
            <div class="title">
                <h1>Flask code</h1>
            </div>
            </div>
        </div>
    </body>
</html>




my form_action.html

<html>
    <head>
        <title>Handle POST requests with Flask</title>
        <link rel=stylesheet type=text/css href="{{ url_for('static', filename='style.css') }}">
    </head>
    <body>
        <div id="container">
            <div class="title">
                <h1>POST request with Flask</h1>
            </div>
            <div id="content">
                Hello <strong>{{name}}</strong> ({{email}})!
            </div>
            <div class="title">
                <h1>Flask code</h1>
            </div>
                <code><pre>
@app.route('/hello/', methods=['POST'])
def hello():
    name=request.form['yourname']
    email=request.form['youremail']
    return render_template('form_action.html', name=name, email=email)
                </pre></code>
            </div>
        </div>
    </body>
</html>


Is this correct?

-tonyphoang 9 years ago

You must be logged in to post. Please login or register an account.

I don't believe I phrased my question correctly.

What I want to do is have a content manager for a folder containing .csv files. Upon clicking on a button (relating to that .csv) it would display the .csv using high charts.

So far I've gotten the .csv file to be displayed as clickable links
my __init__.py

import glob
import re

@app.route('/results/')
def results():
    try:
        data = glob.glob('/var/www/FlaskApp/FlaskApp/csv/*.csv')
        convert = lambda text: int(text) if text.isdigit() else text
        alphanum_key = lambda key: [convert(c) for c in re.split('([0-9]+)', key)]
        data.sort(key=alphanum_key)
        data.reverse()  #reverse sorting
        datanames=[]
        for dirpath in data:
            datanames.append(os.path.split(dirpath)[-1])
        return render_template("results.html", datanames=datanames)
    except Exception as e:
        return render_template("500.html", error = str(e))


my results.html

{% extends "header.html" %}
{% block body %}


<body>

<p>Results</p>

	   {% for c in datanames %}
	   <li><a href="{{c}}">{{c}}</a></li> 
	   {% endfor %}

</body>
{% endblock %}


I'm unfamiliar with how to integrate high charts to open up a .csv file?

-tonyphoang 9 years ago

You must be logged in to post. Please login or register an account.


Highcharts wont open and automatically graph a CSV for you. You have to convert your CSV data to an acceptable JSON. Open in python, convert to a list of lists, then you can convert to json with the json module. Then pass that json object to highcharts to graph.

-Harrison 9 years ago

You must be logged in to post. Please login or register an account.


I'll give that a try and report back.

I would like to try and display my .csv data upon clicking on a button.

my __init__.py

app = Flask(__name__, csv_instance_path='/var/www/FlaskApp/FlaskApp/results_data')

#more code

@app.route('/results/')
def results():
    try:
        data = glob.glob('/var/www/FlaskApp/FlaskApp/results_data/*.csv')
        convert = lambda text: int(text) if text.isdigit() else text
        alphanum_key = lambda key: [convert(c) for c in re.split('([0-9]+)', key)]
        data.sort(key=alphanum_key)
        data.reverse()  #reverse sorting
        datanames=[]
        for dirpath in data:
            datanames.append(os.path.split(dirpath)[-1])
        return render_template("results.html", datanames=datanames)
    except Exception as e:
        return render_template("500.html", error = str(e))


Upon clicking on a link, it produces this a url of /results/#####.csv/

I would like to link the ######.csv to the file. On your Python Flask tutorial #31, I saw something of use but don't know how to link it together.

@app.route('/results/<path:csvname>')
def result_data(csvname):
    try:
        return render_template(os.path.join(app.csv_instance_path, ''),csvname)
    except Exception as e:
        return render_template("500.html", error = str(e))


I also don't know what my .html would look like to pip just the csv into it. Something involving {{}}

-tonyphoang 9 years ago

You must be logged in to post. Please login or register an account.


One option is to load the csv with the Pandas module. from_csv, then you can immediately generate a new object using to_html .... which creates an html table. Then pass that object to your template with the |safe filter, and there's your table.

-Harrison 9 years ago

You must be logged in to post. Please login or register an account.

I've had some success displaying a csv file's content.

__init__.py

@app.route('/results/<path:csv_filename>/')
def display_csv(csv_filename):
    try:
        #connects dirpat of 'results_data' folder to 'csv_filename'
        results_data_dirpath = '/var/www/FlaskApp/FlaskApp/results_data/'
        complete_dirpath = results_data_dirpath+csv_filename

        #reads csv into a list
        open_csv = open(complete_dirpath,'r')
        read_csv = csv.reader(open_csv)
        csv_content = list(read_csv)
        open_csv.close()

        return render_template("converter.html",complete_dirpath=complete_dirpath,csv_content=csv_content)
    except Exception as e:
        return render_template("500.html", error = str(e))


converter.html

{% extends "header.html" %}
{% block body %}
	<body class="body">
		
		<h3><p>The complete dirpath to the csv below:</p></h3>
		<p>{{complete_dirpath}}</p><br>
		<h3><p>The contents of the csv below:</p></h3>
		<p>{{csv_content}} </p>
	</div> 
	</body>
{% endblock %}


I attempted to load high charts on the same code. High charts does not display at all.
__init__.py

@app.route('/results/<path:csv_filename>/')
def display_csv(csv_filename, chartID = 'chart_ID', chart_type = 'line', chart_height = 500):
    try:
        #connects dirpat of 'results_data' folder to 'csv_filename'
        results_data_dirpath = '/var/www/FlaskApp/FlaskApp/results_data/'
        complete_dirpath = results_data_dirpath+csv_filename

        #reads csv into a list
        open_csv = open(complete_dirpath,'r')
        read_csv = csv.reader(open_csv)
        csv_content = list(read_csv)
        open_csv.close()

        #use highcharts
        subtitleText = 'test'
        graphtitle = {"text": 'My Title'}
        chart = {"renderTo": chartID, "type": chart_type, "height": chart_height,}
        Series1_data = [[1,4],[2,5],[3,5]]
        series = [{"name": 'Series1', "data": Series1_data}]
        title = {"text": 'My Title'}                                    #graph title
        xAxis = {"title": {"text": 'x-axis label'}}                     #x axis label
        yAxis = {"title": {"text": 'yAxis Label'}}                      #y axis label
        return render_template("converter.html",complete_dirpath=complete_dirpath,csv_content=csv_content,subtitleText=subtitleText, chartID=chartID, chart=chart, series=series, graphtitle=graphtitle, xAxis=xAxis, yAxis=yAxis)
    except Exception as e:
        return render_template("500.html", error = str(e))


converter.html

{% extends "header.html" %}
{% block body %}
	<body class="body">
		
		<h3><p>The complete dirpath to the csv below:</p></h3>
		<p>{{complete_dirpath}}</p><br>
		<h3><p>The contents of the csv below:</p></h3>
		<p>{{csv_content}} </p>
	</div> 

high charts below:
<div id={{ chartID|safe }} class="chart" style="height: 500px; width: 500px"></div>
<script>
	var chart_id = {{ chartID|safe }}
	var series = {{ series|safe }}
	var title = {{ title|safe }}
	var xAxis = {{ xAxis|safe }}
	var yAxis = {{ yAxis|safe }}
	var chart = {{ chart|safe }}
</script>

highcharts above
	</body>
{% endblock %}



-tonyphoang 9 years ago

You must be logged in to post. Please login or register an account.


Just got it working.

It turned out it was outputting the csv file as strings
[['1','1'],['2','2']]

Just need to convert it to int
csv_content_int=[map(int,x) for x in csv_content]

to convert it to
[[1,1],[2,2]]
which can be fed into high charts pretty easily.

-tonyphoang 9 years ago

You must be logged in to post. Please login or register an account.


Looks like I spoke too soon. I tried to add in the URL converter and high charts doesn't show up.

-tonyphoang 9 years ago

You must be logged in to post. Please login or register an account.


I ended up ditching highcharts for pygal and it's working perfectly.

-tonyphoang 9 years ago

You must be logged in to post. Please login or register an account.