本文介绍python操作数据库的方法,并以mysql与sqlite数据库为例,从一个csv文件中读入数据,插入到数据库中,再将数据库中的数据读出,保存到另一个csv文件。
#介绍
python 社区制定了操作数据库的标准,所以,我们可以通过统一的接口访问不同的数据库,减少了我们的学习负担。
标准主要定义了两个对象,一个用于管理连接的Connection对象,另一个是用于执行查询的Cursor 对象。
#Python 操作数据库的步骤
python 操作数据库的思路如下:
- 导入相应的数据库模块(import sqlite3, MySQLdb)
- 连接数据库(connect),返回一个Connection 对象
- 通过该对象的cursor()成员函数返回一个Cursor 对象
- 通过Cursor对象的execute()方法执行SQL语句
- 如果执行的是查询语句,则通过Cursor 对象的fetchone()等语句获取返回结果
- 关闭Cursor对象(close())
- 关闭Connection对象(close())
#插入数据
对于sqlite 数据库,python 自带了sqlite3 模块,直接导入即可,对于mysql 数据库,则需要安装第三方模块Mysql-python 。安装完以后,在程序中导入模块即可。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 |
<span class="c"><span class="com">#!/usr/bin/python</span></span> <span class="c"><span class="com">#-*- coding: UTF-8 -*-</span></span> <span class="kn"><span class="kwd">import</span></span> <span class="nn"><span class="pln">csv</span></span> <span class="k"><span class="kwd">def</span></span> <span class="nf"><span class="pln">main</span></span><span class="p"><span class="pun">():</span></span> <span class="n"><span class="pln">DATABASE</span></span><span class="o"><span class="pun">=</span></span><span class="s"><span class="str">"sqlite3"</span></span> <span class="c"><span class="com">#open databases</span></span> <span class="k"><span class="kwd">if</span></span> <span class="n"><span class="pln">DATABASE</span></span> <span class="o"><span class="pun">==</span></span> <span class="s"><span class="str">"sqlite3"</span></span><span class="p"><span class="pun">:</span></span> <span class="kn"><span class="kwd">import</span></span> <span class="nn"><span class="pln">sqlite3</span></span> <span class="kn"><span class="kwd">as</span></span> <span class="nn"><span class="pln">db</span></span> <span class="n"><span class="pln">conn</span></span> <span class="o"><span class="pun">=</span></span> <span class="n"><span class="pln">db</span></span><span class="o"><span class="pun">.</span></span><span class="n"><span class="pln">connect</span></span><span class="p"><span class="pun">(</span></span><span class="s"><span class="str">"test"</span></span><span class="p"><span class="pun">)</span></span> <span class="n"><span class="pln">strInsert</span></span> <span class="o"><span class="pun">=</span></span> <span class="s"><span class="str">"insert into stocks values(?, ?, ?)"</span></span> <span class="k"><span class="kwd">else</span></span><span class="p"><span class="pun">:</span></span> <span class="kn"><span class="kwd">import</span></span> <span class="nn"><span class="typ">MySQLdb</span></span> <span class="kn"><span class="kwd">as</span></span> <span class="nn"><span class="pln">db</span></span> <span class="c"><span class="com">#conn = db.connect(host="localhost", user="root", passwd="passwd", db="test")</span></span> <span class="n"><span class="pln">conn</span></span> <span class="o"><span class="pun">=</span></span> <span class="n"><span class="pln">db</span></span><span class="o"><span class="pun">.</span></span><span class="n"><span class="pln">connect</span></span><span class="p"><span class="pun">(</span></span><span class="n"><span class="pln">host</span></span><span class="o"><span class="pun">=</span></span><span class="s"><span class="str">"localhost"</span></span><span class="p"><span class="pun">,</span></span> <span class="n"><span class="pln">db</span></span><span class="o"><span class="pun">=</span></span><span class="s"><span class="str">"test"</span></span><span class="p"><span class="pun">,</span></span> <span class="n"><span class="pln">read_default_file</span></span><span class="o"><span class="pun">=</span></span><span class="s"><span class="str">"~/.my.cnf"</span></span><span class="p"><span class="pun">)</span></span> <span class="n"><span class="pln">strInsert</span></span> <span class="o"><span class="pun">=</span></span> <span class="s"><span class="str">"insert into stocks values(</span></span><span class="si"><span class="str">%</span></span><span class="s"><span class="str">s, </span></span><span class="si"><span class="str">%</span></span><span class="s"><span class="str">s, </span></span><span class="si"><span class="str">%</span></span><span class="s"><span class="str">s)"</span></span> <span class="c"><span class="com">#get cursor object</span></span> <span class="n"><span class="pln">cur</span></span> <span class="o"><span class="pun">=</span></span> <span class="n"><span class="pln">conn</span></span><span class="o"><span class="pun">.</span></span><span class="n"><span class="pln">cursor</span></span><span class="p"><span class="pun">()</span></span> <span class="c"><span class="com">#read CSV file</span></span> <span class="n"><span class="pln">f</span></span> <span class="o"><span class="pun">=</span></span> <span class="nb"><span class="pln">open</span></span><span class="p"><span class="pun">(</span></span><span class="s"><span class="str">"stock_data"</span></span><span class="p"><span class="pun">)</span></span> <span class="n"><span class="pln">stocks</span></span> <span class="o"><span class="pun">=</span></span> <span class="p"><span class="pun">[]</span></span> <span class="k"><span class="kwd">for</span></span> <span class="n"><span class="pln">r</span></span> <span class="ow"><span class="kwd">in</span></span> <span class="n"><span class="pln">csv</span></span><span class="o"><span class="pun">.</span></span><span class="n"><span class="pln">reader</span></span><span class="p"><span class="pun">(</span></span><span class="n"><span class="pln">f</span></span><span class="p"><span class="pun">):</span></span> <span class="n"><span class="pln">stocks</span></span><span class="o"><span class="pun">.</span></span><span class="n"><span class="pln">append</span></span><span class="p"><span class="pun">(</span></span><span class="n"><span class="pln">r</span></span><span class="p"><span class="pun">)</span></span> <span class="c"><span class="com">#create databses</span></span> <span class="n"><span class="pln">cur</span></span><span class="o"><span class="pun">.</span></span><span class="n"><span class="pln">execute</span></span><span class="p"><span class="pun">(</span></span><span class="s"><span class="str">"create table stocks( symbol text, shares integer, price real)"</span></span><span class="p"><span class="pun">)</span></span> <span class="n"><span class="pln">conn</span></span><span class="o"><span class="pun">.</span></span><span class="n"><span class="pln">commit</span></span><span class="p"><span class="pun">()</span></span> <span class="c"><span class="com">#execute statements of insert</span></span> <span class="n"><span class="pln">cur</span></span><span class="o"><span class="pun">.</span></span><span class="n"><span class="pln">executemany</span></span><span class="p"><span class="pun">(</span></span><span class="n"><span class="pln">strInsert</span></span><span class="p"><span class="pun">,</span></span> <span class="n"><span class="pln">stocks</span></span><span class="p"><span class="pun">)</span></span> <span class="n"><span class="pln">conn</span></span><span class="o"><span class="pun">.</span></span><span class="n"><span class="pln">commit</span></span><span class="p"><span class="pun">()</span></span> <span class="c"><span class="com">#close connection</span></span> <span class="n"><span class="pln">cur</span></span><span class="o"><span class="pun">.</span></span><span class="n"><span class="pln">close</span></span><span class="p"><span class="pun">()</span></span> <span class="n"><span class="pln">conn</span></span><span class="o"><span class="pun">.</span></span><span class="n"><span class="pln">close</span></span><span class="p"><span class="pun">()</span></span> <span class="k"><span class="kwd">if</span></span> <span class="n"><span class="pln">__name__</span></span> <span class="o"><span class="pun">==</span></span> <span class="s"><span class="str">'__main__'</span></span><span class="p"><span class="pun">:</span></span> <span class="n"><span class="pln">main</span></span><span class="p"><span class="pun">()</span></span> |
mysql数据库有两种连接方式(可能有很多种),一种是通过指定用户名和密码的方式,还有一种是指定read_default_file 参数,关于.my.cnf
文件,可以参考这里。
#读取数据
下面执行查询语句,并将结果输出到一个CSV文件。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 |
<span class="c"><span class="com">#!/usr/bin/python</span></span> <span class="c"><span class="com">#-*- coding: UTF-8 -*-</span></span> <span class="kn"><span class="kwd">import</span></span> <span class="nn"><span class="pln">csv</span></span> <span class="k"><span class="kwd">def</span></span> <span class="nf"><span class="pln">main</span></span><span class="p"><span class="pun">():</span></span> <span class="n"><span class="pln">DATABASE</span></span><span class="o"><span class="pun">=</span></span><span class="s"><span class="str">"mysql"</span></span> <span class="c"><span class="com">#open databases</span></span> <span class="k"><span class="kwd">if</span></span> <span class="n"><span class="pln">DATABASE</span></span> <span class="o"><span class="pun">==</span></span> <span class="s"><span class="str">"sqlite3"</span></span><span class="p"><span class="pun">:</span></span> <span class="kn"><span class="kwd">import</span></span> <span class="nn"><span class="pln">sqlite3</span></span> <span class="kn"><span class="kwd">as</span></span> <span class="nn"><span class="pln">db</span></span> <span class="n"><span class="pln">conn</span></span> <span class="o"><span class="pun">=</span></span> <span class="n"><span class="pln">db</span></span><span class="o"><span class="pun">.</span></span><span class="n"><span class="pln">connect</span></span><span class="p"><span class="pun">(</span></span><span class="s"><span class="str">"test"</span></span><span class="p"><span class="pun">)</span></span> <span class="k"><span class="kwd">else</span></span><span class="p"><span class="pun">:</span></span> <span class="kn"><span class="kwd">import</span></span> <span class="nn"><span class="typ">MySQLdb</span></span> <span class="kn"><span class="kwd">as</span></span> <span class="nn"><span class="pln">db</span></span> <span class="c"><span class="com">#conn = db.connect(host="localhost", user="root", passwd="passwd", db="test")</span></span> <span class="n"><span class="pln">conn</span></span> <span class="o"><span class="pun">=</span></span> <span class="n"><span class="pln">db</span></span><span class="o"><span class="pun">.</span></span><span class="n"><span class="pln">connect</span></span><span class="p"><span class="pun">(</span></span><span class="n"><span class="pln">host</span></span><span class="o"><span class="pun">=</span></span><span class="s"><span class="str">"localhost"</span></span><span class="p"><span class="pun">,</span></span> <span class="n"><span class="pln">db</span></span><span class="o"><span class="pun">=</span></span><span class="s"><span class="str">"stocks"</span></span><span class="p"><span class="pun">,</span></span> <span class="n"><span class="pln">read_default_file</span></span><span class="o"><span class="pun">=</span></span><span class="s"><span class="str">"~/.my.cnf"</span></span><span class="p"><span class="pun">)</span></span> <span class="c"><span class="com">#crate cursor object</span></span> <span class="n"><span class="pln">cur</span></span> <span class="o"><span class="pun">=</span></span> <span class="n"><span class="pln">conn</span></span><span class="o"><span class="pun">.</span></span><span class="n"><span class="pln">cursor</span></span><span class="p"><span class="pun">()</span></span> <span class="n"><span class="pln">f</span></span> <span class="o"><span class="pun">=</span></span> <span class="nb"><span class="pln">open</span></span><span class="p"><span class="pun">(</span></span><span class="s"><span class="str">"output"</span></span><span class="p"><span class="pun">,</span></span> <span class="s"><span class="str">'w'</span></span><span class="p"><span class="pun">)</span></span> <span class="n"><span class="pln">w</span></span> <span class="o"><span class="pun">=</span></span> <span class="n"><span class="pln">csv</span></span><span class="o"><span class="pun">.</span></span><span class="n"><span class="pln">writer</span></span><span class="p"><span class="pun">(</span></span><span class="n"><span class="pln">f</span></span><span class="p"><span class="pun">)</span></span> <span class="c"><span class="com">#read data</span></span> <span class="n"><span class="pln">cur</span></span><span class="o"><span class="pun">.</span></span><span class="n"><span class="pln">execute</span></span><span class="p"><span class="pun">(</span></span><span class="s"><span class="str">"select * from stocks"</span></span><span class="p"><span class="pun">)</span></span> <span class="c"><span class="com">#write data to csv file</span></span> <span class="k"><span class="kwd">while</span></span> <span class="bp"><span class="kwd">True</span></span><span class="p"><span class="pun">:</span></span> <span class="n"><span class="pln">row</span></span> <span class="o"><span class="pun">=</span></span> <span class="n"><span class="pln">cur</span></span><span class="o"><span class="pun">.</span></span><span class="n"><span class="pln">fetchone</span></span><span class="p"><span class="pun">()</span></span> <span class="k"><span class="kwd">if</span></span> <span class="ow"><span class="kwd">not</span></span> <span class="n"><span class="pln">row</span></span><span class="p"><span class="pun">:</span></span> <span class="k"><span class="kwd">break</span></span> <span class="n"><span class="pln">w</span></span><span class="o"><span class="pun">.</span></span><span class="n"><span class="pln">writerow</span></span><span class="p"><span class="pun">(</span></span><span class="n"><span class="pln">row</span></span><span class="p"><span class="pun">)</span></span> <span class="n"><span class="pln">f</span></span><span class="o"><span class="pun">.</span></span><span class="n"><span class="pln">close</span></span><span class="p"><span class="pun">()</span></span> <span class="n"><span class="pln">cur</span></span><span class="o"><span class="pun">.</span></span><span class="n"><span class="pln">close</span></span><span class="p"><span class="pun">()</span></span> <span class="n"><span class="pln">conn</span></span><span class="o"><span class="pun">.</span></span><span class="n"><span class="pln">close</span></span><span class="p"><span class="pun">()</span></span> <span class="k"><span class="kwd">if</span></span> <span class="n"><span class="pln">__name__</span></span> <span class="o"><span class="pun">==</span></span> <span class="s"><span class="str">'__main__'</span></span><span class="p"><span class="pun">:</span></span> <span class="n"><span class="pln">main</span></span><span class="p"><span class="pun">()</span></span> |
#ubuntu安装MySQL-Python出现mysql_config not found错误
在ubuntu 下安装Mysql-Python时,可能出现”mysql_config not found”错误。提示:
1 |
<span class="typ">EnvironmentError</span><span class="pun">:</span><span class="pln"> mysql_config </span><span class="kwd">not</span><span class="pln"> found</span> |
Google后得知mysql_config是属于MySQL开发用的文件,而使用apt-get安装的MySQL是没有这个文件的,于是在包安装器里面寻找
1 2 |
<span class="pln">sudo apt</span><span class="pun">-</span><span class="kwd">get</span><span class="pln"> install libmysqld</span><span class="pun">-</span><span class="pln">dev sudo apt</span><span class="pun">-</span><span class="kwd">get</span><span class="pln"> install libmysqlclient</span><span class="pun">-</span><span class="pln">dev</span> |
这两个包安装后问题即可解决。
